How to Find Scraped Content (and the Scrapers)



Who's stealing my content? Good question. Find the scrapers with wpCop's aptly-titled investigative guide How to Find Scraped Content.


In How to Protect Content from Scrapers we took steps to preemptively defend our content using a bunch of techniques and plugins. But what if your content is being abused already? This is when we need a reactive response.

We may wish to accept or negotiate fair use and properly attributed scenarios while fighting our corner where our work is being used cynically. First though, we need to know who's using what content, if any. Here's how.

wpCop's WP content security guides

As usual, wpCop's Risk section considers the problem.

Then, we secure our content in wpCop's WordPress section.

So hopefully that's a help. If not, there are forums for each of these topics, as per usual.

Seeking out scrapers

Quite likely, you'll have no clue your content has been collared until you look for copies.

As with most security topics, setting up and mastering a system is what takes the most time but, thereon, keeping tabs on scrapers makes for light work.

We'll trawl the many research options using the web in a moment but, first, let's look at using WordPress and its site logs to help.

  • Investigating the Dashboard
    Oops, hidden content alert!

    Please share for 24 hours nag-free! Thank U 🙂

    Or subscribe for full content, support & no nags!

  • Investigating the site and server log

    Traffic spikes, Digg aside, are often a sign of malicious intent that may be scraping-related. Investigating suspect log entries is something of a trial to begin with but, given a little experience, is actually quite a fast way to uncover a variety of potential threats.

    wpCop explains how to analyze logs (as well as how to manage logs), so there's a clue or two.

    Verdict: Fiddly at first, but important for wider security analysis.

Uncovering plagiarism using the web

Turning over a few digital stones to see who is using what and how may make for a fascinating, if time-consuming, exercise. Fortunately we have tools aplenty and, surprise surprise, many of those involve the vast search capacity that is Google.

  • Searching with Google

    Take a string of text from an excerpt or leading paragraph, wrap it in quotes for precision and run a search. Here's an example:

    “build a fast, secure, future-proofed virtual private server, ideal for multi-sites”

    When you see this message at the bottom of the last result page (and if your phrase is unique, there should not be more than one page), oh boy, click the link:

    Google's omitted results dialogue at the bottom of the search page

    Similar results are precisely what we're looking for with scraped content

    The key words here are entries very similar because those, of course, are precisely what we're after. Tap through the links and, in some cases, remain calm.

    You may also come up trumps with page or post titles, but scrapers often change those.

    Verdict: Excellent use of Google for content producers.

    Don't bother with Google Blogs

    Google Blogs is a cracking service, but not for this job. The thing is, Google Blogs only lists sites with a feed and, typically, scraper sites don't publish feeds. In other words, you won't uncover the crux of the content crackers.

  • Using Google Alerts

    For ongoing surveillance, a less arduous process is to set up Google Alerts to inform you every time a keyword is used. Signed into your Google account, register a phrase or six:

    the Google Alerts setup panel

    Setup alerts for unique site phrases and keywords

    Good examples include your name, business name, Twitter handle, domain name, blog title and a much-used phrase such as we used above. A phrase is particularly efficient for deep scanning because it's less likely to be changed by a scraper than is, say, a reference to your site.

    Embed a regular signature catchphrase, or two, directly in your WordPress post content and also in your feeds using a plugin like ©Feed, that wpCop introduced in Protect WordPress content from scrapers. Then set up alerts.

    Verdict: Super-nice tool, use Alerts.

The big G aside, here are some more avenues of online detection.

  • Copyscape

    Submit the address of a webpage, and Copyscape's Google-powered engine scans for copycats, whether for full or partial use of your content:

    Copyscape's search input field

    Chasing scrapers using Copyscape

    The basic service is free, but limited to a very few searches per domain, per month. The problem is that our time is also limited. The premium service, Copysentry, comes in here, continually scouring-devouring, and you may find better value with that. Copyscape's other service, Copyscape Premium, is the one to use if you employ unproven writers, scrutinizing their text to ensure its originality:

    Verdict: Many swear by Copyscape. Copysentry makes time-saving sense for big sites.

  • Feedburner's Uncommon Uses

    If you use Feedburner to manage feeds, then this feature helps track suspicious use. Click on Uncommon Uses in the Feed Stats on a feed's main panel and, discounting known false positives, paste suspect URLs using Google parameters such as these:

    site:suspectsite.com “optionally add a signature catchphrase in quotes as well”

    You can combine other special Google search syntax as well, as the ‘Cop details in Google hacking.

    Verdict: This is powerful both to flush out copycats and to gauge brand penetration.

  • Plagium

    Plagium seeks out the contagion that is plagiarism. (Sorry.) You can submit a page, set up alerts and, superbly, scan against large chunks of copy.

    Plagium offers a top-notch compliment to Google analysis because, apart from offering highly featured value and unlimited hits for absolutely free, this service is Yahoo-powered, rather than Google-powered, helping us to cover the bases:

    Verdict: Worth using, at least for badly scraped sites.

  • TinEye

    TinEye checks for duplicate images. Just upload a picture or submit a URL and, within seconds, well over a billion images are scanned and a list of duplicates is created:

    Verdict: Highly advisable for snappers.

Pinpointing scrapers

When you've found an offending scraper site you can start a dialogue … or you could if there was one of those new-fangled Contact Us buttons. Often, there aren't. As for the About Us button, scraper types are often shy of those too. An e-mail address? Hahaha!

You could chance it and simply send an e-mail to webmaster@somescrapersite.com. Then again, it's better to be precise, if only to show that you know your way around the web. That way you will be taken more seriously.

Run a WHOIS search

Here's a clue. To save repetition, read WHOIS whacking which explains how to find the right service to gain the most detail for the offending domain. Unless the domain registrant has applied privacy settings, the results will yield many details to get you started. For example:

  • The name and address of the domain registrant, or owner
  • The domain registrar
  • Administrative and technical contacts, addresses, and telephone numbers
  • The domain's IP address
  • The nameservers, giving a clue as to the web host

From the record you will, at least, be able to track contact details for the domain's ISP by taking a nameserver and running it by a service such as Network-Tools. The resulting DNS records will provide, among other things, e-mail addresses for the host's support and, generally, for their abuse departments.

From investigation to acting on scrapers

Cool, we've uncovered whatever instances of content abuse and we know who to talk to.

So what now?

In Deal with Content Scrapers: Multiple Techniques we'll look at a variety of methods to employ to regain our work – from meow to snarl to bark and bite – and edit some proven stock template letters that we can issue.