A SQL DBA tries NoSQL – Part 3 – Redis (Key-Value)

The purpose of this exercise is to get a little familiarity with a range of NoSQL technologies. See my learning plan for some context.

Redis seems to be the current prevailing key-value technology. I thought I would have to download and install redis to play with it, but there is a great interactive tutorial on the redis website which provides an introduction to how developers will be using the technology.

It naturally starts with the commands to set and get a key value, and in doing so, must introduce the concept of atomic versus non-atomic operations, and wrapping a transaction around a series of commands. I wanted to read more on transactions, and got the lowdown on transacations here.
This little nugget jumped out at me: “if the Redis server crashes or is killed by the system administrator in some hard way it is possible that only a partial number of operations are registered. Redis will detect this condition at restart, and will exit with an error.

Oh. So you have to explicitly do something about this?  Why doesn’t it roll back? Seems strange.

Then all became clear a few paragraphs later:

If you have a relational databases background, the fact that Redis commands can fail during a transaction, but still Redis will execute the rest of the transaction instead of rolling back, may look odd to you.

Well, you took the words right out of Keanu’s mouth.

 

The redis website offers two reasons for not supporting rollbacks. One is perfectly understandable: “Redis is internally simplified and faster because it does not need the ability to roll back.” I get it. You compromise on one area to improve another (seems a mighty big compromise, but that’s another story).
Their other reason is less understandable to me, basically errors won’t happen (much). “in practical terms a failing command is the result of a programming errors, and a kind of error that is very likely to be detected during development, and not in production.” Nope, not really buying that. But I’ll buy the fact that usage must be done with understanding and acceptance of the compromise being made.

Jumping back to the tutorial, it takes us through setting an expiration date on a key, and then moves on to commands for interacting with lists and sets. Then it gets into hashes, which basically can represent an object and its multiple attributes.
At which point it wraps up. It would have been useful it the tutorial had given an indication of progress i.e. I was wondering where I was with it and was hurrying a little. But it only takes about 30 minutes to complete, with a little bit of playing and experimenting.

Having played with it, I looked through a few articles for an overview and commentary, and took away some “key” points (heh, heh) which make it stand out from the crowd. This was a good overview, from a “web astronaut” no less.

In-Memory

The database is in-memory. A corollary of this is that it won’t support a data set that is larger than the memory space (i.e. by persisting overflow to disk). Basically, its not for large data, and is optimized for fast read/write of small textual information.

So how does Redis deal with the server shutting down? There are two forms of secondary storage, RDB and AOF. At specified intervals, Redis snapshots to its data store, the RDB file. Typically, this might be every five minutes, so still brings data loss into the equation in the event of shutdown. The SAVE command will also persist a snapshot to the RDB file. There is also an AOF file, which gets operates through all writes being persisted to this file. This is akin to the transaction log in SQL Server. The general recommendation is to use both methods if data loss is unacceptable.

Backups

There is no proprietary backup mechanism, so the administrator needs to work out a mechanism to do this e.g. file copy of that RDB file via a cron job.

Replication

However, fault-tolerance is provided through replication of data to slaves. Redis replication is full replication, all slaves contain the same data as the master. By adding a slave, it is automatically synced from the master. If a slave fails, when it comes back online the master will sync to it. If the master fails, then a slave can be converted to being the master. Replication is also to support performance, as the master will distribute large reads and sorts for parallel execution.

Configuration

How is it configured? Once installed, there is a configuration file (redis.conf) at the root directory. This can be edited directly, or there is a CONFIG command that can set all configurations e.g. loglevel, maxmemory, maxclients.

This was an interesting read on configuring for production. My main take-away was on how this thing should be monitored.

Monitoring

How does monitoring work? There is a log file that may be the first point of call. Other than that, the administrator should set up a monitoring service of some kind. There are several open source utilities which are installed as a service and will poll the status of Redis and provide alerts for RAM or CPU thresholds. They can also be configured to restart the redis server it is stopped.

 

 

 

 

 

 

eBay Selling: “Tracked” and “Signed For” Delivery

It’s easy to get confused between the different ways that postal delivery is described by your chosen delivery service, by eBay and also by PayPal. But it’s important to know what you are offering your buyer, as there are crucial differences in what protection you may have if the buyer is seeking a refund.

Tracked Delivery
Let’s start with “Tracked Delivery”. With this method, the postal service provides you with a tracking ID that shows you the progress of the item. Usually, the postal service will have a website area that allows you to enter the tracking details and view the status of the item – this could be at Sorting Office A, in transit to Depot B, or what you want to see most: “Delivered”.
Now, it’s important to realize that “Delivered” isn’t a guarantee that the item has been placed into the grateful hands of the buyer.

As many people report, depending on various levels of postal service, the item could have been chucked onto the porch, or handed to a helpful neighbor. The postal worker dutifully records the item as delivered.

The well-meaning buyer may well fall victim to items disappearing from their porch, or the neighbor taking off on vacation that day and forgetting all about that package they put into their closet.
In other words, “Tracked Delivery” may not mean “Signed For By the Buyer.”

This may not matter to you. As of this time of writing, if the item was paid for with PayPal and meets certain other requirements then you have “seller protection” if the item was sent with “Tracked Delivery”. In the event of the Buyer claiming they did not receive the item, you should not be required to refund the money. One of those requirements is the value of the goods – this threshold changes, so be sure to check what cover you have by going online to eBay or PayPal’s help center.

Signed For Delivery

If using PayPal, you can avail of one more level of protection: “Signed for Delivery.” As I mentioned, if the value of goods is over a certain amount, then “Tracked” won’t be enough. “Signed for Delivery” gives both tracking details and an additional level of proof of a signature that the goods have been received. It will usually be the most expensive method of posting. It is also the most onerous and annoying for the Buyer – they are either hanging around waiting for the goods, or they have to schlep to some postal depot at the far side of town. Many sellers avoid “losing” buyers by restricting to this layer of protection. This is your judgement call on the risk of loss due to an INR case.

So is “Signed for Delivery” the absolute and final guarantee that a Buyer wont’ try to claim INR? Unfortunately not. Think of the last time you “signed” for a delivery using a scanner – you tucked the item under your arm while balancing the hand-held scanning device against the side of the delivery van, and scratched awkwardly with a stylus pen a few scrawls thjat bear some semblance of your name. Some sellers are experiencing buyers who simply claim that they didn’t provide a signature and they don’t recognize the version the postal service has recorded.

Here is one case:

“The item was duly posted off the next working day via [company name removed] with tracking to the buyer’s business premises. The buyer later informs me he has not received although the tracking details give an exact time of delivery. However, the buyer claims he does not recognize the signature provided. [Postal company] are adamant the item was correctly delivered and thus refuse to accept a ‘lost or damaged’ claim.”

If PayPal was not used as the payment method, then it is with eBay representatives to adjudicate on a Buyer claim. If eBay side with the Buyer, then the Seller has to resort to get compensation for a refund from the delivery company. If the delivery company digs in their heels, as in the example above, unfortunately the Seller may need to resort to litigation.

A higher level of fraud can occur, simply because the delivery company does not look for additional identification from the person receiving the item. It’s possible for scammers to sign a different name, and claim INR. However, this is upping the stakes from a “casual” scam to a new level of fraud. It is likely in this event that the person is a serial scammer, and may already be “of interest” to the delivery service, or to local policing authorities. All the more important for the Seller to report the details to the delivery company.

However, although INR cases are widely reported, the scenarios of Buyers disputing Tracked and Signed For Delivery is far less prevalent among the online forums.

In most scenarios, when a Buyer opens a case, you will simply upload the tracking details and let eBay sort it out.

It’s actually preferable to cut this situation off before it gets as far as a case (which goes onto your selling record). When a buyer contacts you saying they didn’t receive the item, send them the tracking details. That is often enough for the item to “turn up” the following day.
Casual scammers are likely to be sending the email in the hope that the busy Seller will simply click the Refund button and move on. Getting the tracking details quashes their attempt straight away. Many experienced sellers have a routine of posting the item followed by emailing the tracking details, same day, to the Buyer.

In summary, what should you do if you are covered by Tracked Delivery and a buyer claims INFR?
If they have contacted you informally (haven’t opened a case), be courteous and email them the tracking details. Then follow the advice given in previous blog entries about checking with neighbors and family etc.
If they open a case, upload the tracking details immediately and let eBay’s process kick into place.

<u><em><strong><a href=”http://eepurl.com/cAz9Zb”>Launching Soon: “eBay Selling: Avoid Scams and Fraud when Selling Goods Online”</a></strong></em></u>

 

eBay Selling – is Proof of Postage worth the paper its written on?

For Seller protection against “Item Not Received” fraud, Proof of Postage is the next level up from no protection at all. You go to a post office and get a stamped receipt that acknowledges you handed a package over to the postal delivery system.

Of course, this is of no use to the buyer. When issues occur in transit, the genuine buyer will probably be even more frustrated if you point out repeatedly that you posted the item.

If the buyer states that they did not receive the item and requests a refund, online marketplaces such as eBay will probably take no account of POP. They will likely refund the buyer if that’s all that you’ve got.

Confused sellers regularly pop up on forums wondering why proof of postage isn’t enough to avoid refunds. Remember, POP is a contract between the SELLER and the postal delivery service, it has nothing to do the with the buyer.

So what good is proof of postage? To be honest, not a whole lot.

Benefits of Proof of Postage for the Seller

The main benefit is that you may make a financial claim against the postal delivery service. Check the terms and conditions of the company – there is usually a cap on what you can claim.  There also may be gotchas and caveats in terms of what you can claim. There are some items that are not supposed to be sent POP anyway – perfume and other liquids are not covered in certain countries.

Due to the cap used by your delivery company, the amount you can recover from a claim may be a small percentage of how much you’re out of pocket by a refund to the buyer.
If the loss is unacceptable, then move up to the next level of protection: recorded delivery.

There is a “hidden” benefit that won’t help you immediately, but does contribute to making online selling less fraught with fraud. If you just throw up your hands and write off the refund without making a claim against the courier, there is little extra deterrent for fraudsters. But repeated claims recorded against the same target address will feed into the courier’s own protection systems. You probably will never know about it, but accumulated claims may result in further investigation.

Another benefit has nothing to do with INR, but I mention it here for completeness. POP may give you protection against a chargeback claim of unauthorised card use, which can be made up to a year after payment. Successful chargeback claims can rack up fees on your account.

Aarggh, the Buyer is claming INR

Okay, so you took a chance on POP and the buyer is saying they didn’t receive the item. It’s important to keep all communication courteous. Do send a scan of the POP receipt to the buyer, but don’t try and make it seem like that lets you off the hook.

Ask the buyer to check with other members of the household and with neighbors in case they took in the item.

If no joy, ask the buyer to check with their local postal office. When items can’t be be delivered (e.g. they won’t fit through the letter box), the postal service is supposed to leave a notification – these can get lost under the carpet, be eaten by the cat etc.

If you intend to make a claim, check with the postal service as to their time lines – it could be 15 or 20 working days before they consider a claim. At this point, you may follow up your other courteous communications with the Buyer, and ask them to wait until it is deemed “officially” lost before they look for a refund or replacement.

Finally, you may actually need the willing help of the Buyer to make a successful claim. The postal service may contact the buyer to ask them to confirm non-receipt – if they don’t respond, the compensation is withheld. Yes, it sucks for a seller.

Launching Soon: “eBay Selling: Avoid Scams and Fraud when Selling Goods Online”

A SQL DBA tries NoSQL – Part 2 – The Basics

“Key-Value”, “Document”, “Wide Column”, “Graph”, “Search” – these are the terms used to categorize a very long list of technology offerings under the heading of NoSQL. I think if I was shown a diagrammatic representation of data for each, and was offered a million dollars to label them correctly – I could do it. But it’s worth taking a look at the basic tenets before jumping into specific technologies.

All I’m looking for now is to give myself a basic mental model of data storage. A few simple images for a quick refresher when I return from wrestling with SQL Server replication conflicts.

Key-Value Store

Database.Guide has a nice explanatory article, with simple visual examples. Here’s my own example, using a list as the value. There are two disparate sets of data here – one describing NoSQL technologies, and the other describing blog posts.

So no foreign keys, pardon me while I find my smelling salts. No defined schema, I may not recover.

The Database.Guide article mentions Redis as top of the list of Key-Value technologies. A little way down that list…I promised myself not to fall down rabbit holes, but…Voldemort…I can’t resist following the link for a peek at He Who Shall Not be Named.

Document Store

Okay, I’m back. Now to peruse Document Stores. This Database.Guide article is clear, and gave me a chuckle. The Wiggles! And here’s my own example:

Hmm. After reading first about Key-Value stores and then about Document stores, I’m having a little trouble seeing what’s the difference. I mean, a document store seems to be a key value store with tags? Reading around a bit more…well, I’m not altogether wrong.

Okay, so document data can be organized into collections e.g. Customers, Orders, Products. These collections can be partitioned and indexed and all kinds of other interesting activities to give performance benefits for querying.

So when to use which? This seems a good breakdown: “if you usually retrieve data by key or ID value and don’t need to support complex queries, a key-value database is a good option…If you have different types of entities and need complex querying, choose a document database.

Column Store

Or “Wide Column”? Or “Columnar”? C’mon, community, pick one, ‘cos it’s confusing.

For some reason, I don’t find the Database.Guide article as explanatory as the others. Here is an alternative, which is a quick read with a good graphic. The actual product described in that article offers both row-based and column-based features, so the side-by-side was useful.

This is another good article. So basically, we’ve got columns instead of rows. Got it.

Main points I’m taking are that each attribute is stored in it’s own file or memory region, which gives faster queries on specific files, and allows greater compression (less variety in the data).

Graph Databases

Whew, so I finish up with a quick overview of Graph databases. Most articles on the Net seem to be from Neo4j, let’s see if I can find something from somewhere other than the leading vendor.

No pics in this old-skool white-on-black page, but it’s a good short write-up.

However, I feel that I will need some passing familiarity with graph theory to really get my head around this. I enjoyed these two short-ish videos from mycodeschool.com:

 

 

 

A SQL DBA tries NoSQL – Part 1 – A Learning Plan

I’m a relational DBA looking to get a grasp of some of the NoSQL database technologies knocking about.

There are so many implementations that it’s hard to know where to start. MongoDB or Neo4J or Cassandra? I don’t want to fall down the rabbit hole of of one particular technology only to find that it’s solely used by the developer’s mother, or that it falls by the wayside next year (RethinkDB *)

First thing’s first – what are the most popular new technologies out there? There’s a lot of possible metrics for this, DB-Engines is as good as any when we’re thinking of usefulness to our careers.

The good news for me is that my main expertise is in the Top Three. Well, I’m not interested in Relational DBMS for this purpose, so I’ve decided to exclude those and focus on the top-of-the-pops for some of the other categories presented: Document Store, Wide Column Store, Key-Value Store, Search Engine, Graph DBMS.

In terms of categories, I’m ignoring Time-Series (InfluxDB), Content Store (Jackrabbit), RDF (Jena), Object-Oriented (DB4o). Why? Because their top DB is way down the list, well-outside the top 20. I could also ignore Graph DBs because it’s #1 is outside the top 20 (actually it’s 21 at time of writing). But I’m not a total empty vessel – an employer once asked me to take a look at some alternative graph offerings. That’s good enough for me to pull them into my list – one commercial enterprise I know was thinking about them!

Here is the DB-Engines top 30, excluding Relational Engines. There’s no easy way to filter out “all but one” using their site, so you’re welcome.

For my own personal “to-do” list of databases to take a look at, I’m going to add a few that are way down the list, because I’d like to have at least two per “type”, and for some – they’ve come up in work conversation, albeit with developers moon-lighting in mobile apps – again, a little familiarity gives me some notion that they are being used at least by someone in the city in which I’m likely to work.  So here’s my target list of technologies to get familiar with:

As it happens, I’ve worked extensively with one of these, but the others are fresh and new. Now, when I say I worked extensively with a technology – I didn’t do much in the way of hands-on administration or scripting. I worked on export and transformation of data from an RDBMS, providing large data-sets to another team who specialized in ingesting the data into the non-relational technology. I had to talk with those guys, trouble-shoot our interfaces, work late and joke around with them, good times.

I’d never seen the technology so I approached it the way most do, I guess. I carved out some time to download the software and work through some tutorials on installation, maintenance, import of data, and usage of same. That was enough for me to:

  • maintain a shared solution for the secure and stable flow of data from RDBMS to non-RDBMS system
  • decipher the complaints of the developers when half the data goes “missing in transit”
  • bring coffee and sympathy to the non-RDBMS admin when his replication…ahem..doesn’t

 

That’s the level I want to approach with my target list. I’m looking forward to playing with new DB toys, but I don’t have time to mess about with thirteen technologies. I’ll get my hands dirty with MongoDB, Cassandra, Redis, ElasticSearch and Neo4J.
For the rest, I’ll browse a few articles. Then set myself up as a consultant (joking).

So there’s a learning plan. But before I actually dive into one of these things, there’s something I need to take care of. “Document”, “Wide Column”, “Graph”, “Search”, “Key-Value” – can I define the main characteristics and differences between each of these terms? Well, yes, kind-of. Hey, I’ve picked up a few things over the years. But then again, no – they haven’t put food on my table, so I’m not exactly on firm ground here.

So I’m starting with a basic overview, in the next post in this series.

(*) A note on my reference to ReThinkDB. I don’t mean it as a slant on the technology. The link in the first paragraph is to one of the most honest and illuminating articles I’ve read on a group of people trying to make something really good.

eBay Selling – “Item Not Received” Fraud

The “Item Not Received” scenario occurs when the buyer claims they never received the goods. They either request a refund or the seller will send a replacement item.

The benefit of fraud for the buyer is that they get the goods and their money back (i.e. they’ve got the item for free), or they get two items for the price of one.

HONEST COMPLAINTS

We must acknowledge that the honest buyer is suffering here from the frustrating experience of items lost in the postal delivery service. They legitimately expect sellers to be considerate, recognize their disappointment, and try their best to help.

In the United Kingdom, the Royal Mail reported that in the financial year of 2010-2011 “we received about one complaint for every 13,000 items of mail we delivered.” That is a tiny percentage.

It’s more difficult to get figures for the United States postal system. The USPS is up-front about the amount of mail they could not deliver due to addressing issues, which in 2010 was 4.7%. We’re talking about indecipherable addresses, weather-damaged envelopes, and blank printed labels. Now, as a seller, you presumably take care in getting the addressing right, and are more interested in the percentage that falls into a black hole due to system or employee error. I spent some time searching for statistics of customer complaints to no avail.

Similarly, Canada Post will not reveal exact numbers of complaints about missing post.

If you’re wondering why the Royal Mail was more open as recently as 2011, the company was a public service for most of its history. The ordinary citizen could get these statistics as part of the Government Freedom of Information Act. Their counterparts elsewhere insist that such data is commercially sensitive information. In 2015, the company became fully privatised.

So let’s put a range of 0.01% to 3% on our estimates of lost mail. Online sellers of physically delivered goods should pick a reasonable number and cost it into their business. If you are sending out 100 items a month, you might factor in having to replace or refund three of those items. If you’re moving house and doing a once-off sale of ten items, you might think you’re unlikely to be unlucky enough to have one of them go astray.

OPTIONS FOR POSTAL DELIVERY
When sellers use the postal service to send goods, their choices are
1. Send through standard mail, no proof of postage, no recorded delivery
2. Obtain proof that the seller has posted the item, usually by going to a postal office or depot and getting a stamped receipt
3. Obtain proof that the buyer has received the item, by using a recorded delivery service

Let’s go through these one by one, starting with…

PERILS OF USING STANDARD POST
There are many reasons to go with standard post.

  • It’s the cheapest method of postage.
  • Proof of Postage is awkward for sellers, it requires physical travel to postal offices that offer the service, this may be onerous in many locations
  • Recorded delivery is awkward for buyers, they have to be around to accept delivery, or travel to a pick-up depot. Some buyers may simply choose to avoid trading with sellers who insist on recorded delivery
  • The seller is a crazy gambler

 

The seller has no protection if the buyer makes a claim to the online marketplace that they did not receive the goods.

That’s worth repeating: the seller has no protection if the buyer makes a claim to the online marketplace that they did not receive the goods.
 

If you are selling low-cost items in high volume, then you know your business and you absorb the risk.
If you are selling within a local marketplace where you know the buyers personally, then you may choose to rely on trust and reputation.
If you are not doing either of above, you’re a crazy gambler to stuff that smartphone into a padded envelope and drop it into the nearest post box.

So let’s say you acted like a crazy gambler, and you’ve received an email from the buyer, loaded with exclamation marks, saying they haven’t got their item yet. Or without even an email, you receive notification from the marketplace that the buyer has requested a refund due to “item not received.”

If you suspect that the buyer is pulling a fast one, there are a few last-ditch tactics you can try. I’ll describe these in another post.
Ultimately, you’ll probably have to accept you rolled the dice and lost.

  • Send a refund
  • Keep a record of the buyer ID
  • Keep a record of any email they sent (or call they made) (I’ll detail why in another post)

Launching Soon: “eBay Selling: Avoid Scams and Fraud when Selling Goods Online”

Email Spoofing – why it’s still around and how to spot it

In the previous post, we discussed one mechanism for authenticating the sender of emails. As spoofing is still a wide-spread problem, why hasn’t SPF solved the problem? Let’s look again at Email Headers, a standard set of information that is transferred with every email.

 

In the usual display of most email clients, we see a subset of the header.

 

This time, we want to look at the full header – this is achieved in different ways, depending on your email client. In Outlook Mail, hunt around for “View Message Source”. In Squirrel Mail, the option is easier to find: “View Full Header.”

Good Sender

Let’s look at an email from Amazon, I’ve eliminated some of the lines for easier reading, and bolded the important lines for our purposes.

Received: from VE1EUR02HT210.eop-EUR02.prod.protection.outlook.com
smtp.mailfrom=bounces.amazon.com; hotmail.com;
dkim=pass (signature was verified)
Received-SPF: Pass (protection.outlook.com: domain of bounces.amazon.com designates 54.240.15.125 as permitted sender)
Date: Wed, 18 Jan 2017 20:04:13 +0000
From: “Amazon.com” <store-news@amazon.com>
To: redacted

Now you’ll see that email messages actually contain two sender addresses. The “MailFrom” address, and the “From” address. Only the second, the “From” address, is displayed by an email client. The first is there, but hidden from plain view.

Crucially, SPF only checks this hidden “MailFrom” address. Legitimate senders will often want to modify the displayed From address, as discussed in the previous blog post.

Notice in my example that the receiver did its due diligence on the SPF record. The sending server’s I.P. address is 54.240.15.125. The receiver grabbed the domain from the “mailfrom” and checked its list of authorized servers: “the domain of bounces.amazon.com designates 54.240.15.125 as permitted sender

Bad Sender?

Now I’m going to show you another mail header for an email I received to my Outlook mail address (I’ve changed some details for privacy).
The receiver gets the domain from the “MailFrom” address and goes looking for its SPF record listing the authorized senders.  In this case, the host (lbntechnology.com) hasn’t set up any SPF record (“lbntechnology.com does not designate permitted sender hosts”).

Received: from VE1EUR01HT052.eop-EUR01.prod.protection.outlook.com
smtp.mailfrom=lbntechnology.com; hotmail.com;
dkim=none (message not signed)
Received-SPF: None (protection.outlook.com: lbntechnology.com does not designate permitted sender hosts)
Date: Wed, 25 Jan 2017 21:59:14 +0000
From: Casey <casey@lbntechnology.com>
To: redacted

Remember, Microsoft were early adopters of SPF. Does Outlook reject this message? No, it flags in the header that it hasn’t got a “pass”, but merrily displays it to me. Only if I look at the message source do I see that the SPF verification could not take place. There was no spoofing attempt here, just an administrator of a domain that doesn’t bother setting up SPF records.

So, in this case, Outlook is allowing domains that don’t “sign up” to SPF to bypass rejection. Knowing this, spoofers will tend to look for domains that do not publish SPF records. It’s perfectly possible to increase the severity of spam filters to reject non-compliant domains – this is known as a Hard-Fail. The problem is that huge amounts of otherwise-legitimate emails would be bounced.

Fatal Flaw

This is the fundamental blocker for SPF or any other authentication protocol. Without global take-up of the protocol, mail providers are reluctant to reject emails that don’t bother with authentication at all.

The good news is that although we can be fooled by the displayed FROM address, we now know a way of checking the sender’s domain.

Guarding Against Spoofing

 

Using your email client, take the trouble of viewing the message source and looking for the “mailfrom” details.

It appears near the beginning of the output, before all the extensive gobbledygook that comes afterwards.

With a bit of experience, your eye will jump straight to the line.

 

Watch Out – Homoglyphs About!

One extra hurdle from savvy scammers is the use of “mailfrom” domains that resemble the real thing. So you’re diligently looking for paypal.com, and the actual text is “paypa1.com” i.e. the same except for one similar looking character.

In this case, the scammer has registered the paypa1 domain, and is merrily sending spoof emails from their mail server.
This is known as a homoglyph attack – where the text looks the same to the casual eye. One precautionary measure is to copy the text from the message source and paste it into a document with a larger font. Yes, you’re probably not going to do that with every email – but if you’re selling an item for $1,000 then the extra few seconds are worth the hassle.

 

Email Spoofing – faking the Sender

The previous blog post explained how the scam of fake PayPal notifications played out from the point of view of the victim, the Seller of goods on a online marketplace.
The seller receives an email, which is faked as “sent” from a legitimate looking address such as “service@paypal.com”.

This is the email header of a PayPal notification as displayed in my Outlook Mail:

 

 

Here is the same header displayed in SquirrelMail:

 

 

 

Outlook displays a fancy logo beside the FROM address, while the more basic SquirrelMail doesn’t mess around. Either of these emails could be spoof PayPal emails, it’s impossible to tell from what you see in the standard email display.

This post examines how scammers go about faking PayPal notification. I’m going to go into the underlying technical details, for those who are interested. Unfortunately, I have no fear that I am “revealing” to wannnabe scammers how to go about their business. To run the scam, you don’t need to know how to set up a system that sends spoofed emails. There are many websites out there that will give the lightweight scammers a simple interface to send a few emails a day to eBay or Amazon sellers. They enter a few details into a web page: the “fake sender”, the recipient, and the body of the message – and away they go.

Want to try it?

Search for “free online email spoofer” and you’ll get a ton of websites offering to do the nasty deed. Click on one of them and you’ll probably pick up a virus, and serve you right. Lie down with dogs, and all that.

Try enough of the sites, and you’ll find one that just insists that you watch a load of dodgy adverts in order to use their service.

So, behind these websites, there is a mechanism that allows email “sender” details to be spoofed. I mean, when you’re using a website called “www.dodgygeezer.ru”, that domain shouldn’t be allowed to send emails purportedly from TheDonald@whitehouse.gov, right? (Rhetorical question, folks, the answer is no, it should not). So how do they do it?

The larger-scale scammer has access to an email server, otherwise known as an SMTP server. They may have hacked an otherwise legitimate server, but it’s perfectly possible to pay twenty bucks a month to a web hosting provider for a virtual server on which they can install the SMTP mail software of their choice.

By the way, it takes not much more than a day to set up an SMTP server from scratch and configure it properly. Most of that time is waiting for the new details to be recognized across the world wide web. Installing and configuring the software itself takes under an hour.

Now, some mailing software insist on defaulting the Sender details to one or more accounts registered at a specific domain (e.g. service@dodgygeezer.ru). This is usually the domain associated with the web server provided by the hosting provider as part of the package. But some perfectly legitimate mailing software freely allows the entry of any “FROM” address – mickeymouse@disney.com is no problem at all.

Why is this legitimate?

Well, many businesses like to help their customers by segmenting their email output. Instead of every email coming from “service@goodbiz.com”, depending on the reason for the email, it may be modified to come from “returns@goodbiz.com” or “helpdesk@goodbiz.com” or “customersurvey@goodbiz.com”. These email accounts may not actually exist – when the recipients hit reply, the messages all return to the same generic recipient, allowing internal software to parse and route the message as appropriate (carefully losing the complaints).

So, the folk at goodbiz.com are faking email addresses from their own domain. They are not trying to dupe their customers, and the software they use facilitates their customer interaction model.

But surely it’s reasonable for the internet-using public to expect that goodbiz.com should not be allowed to send emails with the sender specified as @disney.com or @paypal.com. The mailing software may allow it, but isn’t it possible for these emails to be intercepted as illegitimate?

But surely it’s reasonable for the internet-using public to expect that goodbiz.com should not be allowed to send emails with the sender specified as @disney.com or @paypal.com. The mailing software may allow it, but isn’t it possible for these emails to be intercepted as illegitimate? 

Intercepted by which slavering guard dog, you ask?

By the hosting provider. They maintain all network traffic sent from the servers they lease to the public. They “see” all those mail headers, they see the “SEND” details and they know the domain from which the email is originating.

Wait, our friends at GoodBiz have an objection. They follow good principles of system safety, and run their mail software from a different server than their website. Not only that, but GoodBiz has a subsidiary company called SmallBiz which shares its parent’s mail server. And the SmallBiz help desk want to send emails from the main GoodBiz support account.

Okay, our fraud interception policy needs to be extended in two ways.

Firstly, we must allow website owners to nominate other servers that they lease to be part of a family of servers that are associated with their domain.

Secondly, we must allow website owners to nominate other domains that they own that are allowed piggyback on the SEND details of a specific domain that is part of the family.

In other words, the domain owner must specify which Senders are Permitted to send From mail servers registered to their domain. We’ll call it SPF for short (Senders Permitted From).
This allows either the hosting provider or the receiver to look at the email header, and make a call to the domain asking if the SEND details are on the Permitted List.

Wow, this is radical new thinking, brought to you here first on this blog!

 

Actually, no.

Let’s go back to the middle ages (in internet time).

Way back in 2003, Singaporean entrepeneur Wong Meng Weng devised this strategy, originally called Sender Permitted From. As others got involved, the name changed to Sender Policy Framework. Weng and others put great effort into evangelizing this framework to fight spam.

Crucially, Microsoft were working on something similar, and broadly accepted the approach. In 2005, they rolled out an implementation of SPF into their mailing software.

So why, in 2017, are eBay and Amazon sellers receiving spoof emails from paypal.com?

Next blog post coming soon.

eBay Selling and Fake PayPal Notification – How It Works

The Scam

Payment notification fraud takes place when the Seller of an item receives an email falsely stating that the buyer has paid for the item. The email is faked to look as if it comes from PayPal or the Online Marketplace (e.g. eBay or Amazon).
The sender will look official (e.g. payments@payal.com), and the email will contain authentic-looking logos. Often the subject and body text will be very similar to the official text that a Seller is used to receiving.
The Seller ships the item to this scammer, only to discover too late that they haven’t been paid.

Plus Phishing

fishingSimply sending unpaid items is possibly the “least worst” outcome of this scam.

The fake email may go to some lengths to entice the Seller to click on embedded links or graphics in order to “re-submit” their PayPal or marketplace details.

This link will take the Seller to a webpage that looks like PayPal or whatever, but is in fact a phishing page, designed to obtain your username and password for further malicious theft.

 

 

Telltale Signs

There are some other characteristics to watch out for as this scam plays out.

warning

If the Seller usually takes payment direct to their bank account, the Buyer will insist that PayPal is used. Why? Because PayPal genuinely sends email notifications of payments – your bank account does not, so you’re more likely to notice that something is wrong here. There are reports of scammers sending email purporting to come from *their* bank saying that money has been transferred to your account. This is less popular with scammers, as sellers may be more suspicious of, what is for them, an unusual message, whereas they are quite used to PayPal notifications.

Many sellers report that the buyer explicitly asks for their PayPal email, or for a PayPal Money Request sent to their own email address. This is a red flag, as it is moving the transaction outside of the online marketplace trading and messaging system. The scammer wants the PayPal email because Sellers are accustomed to receiving funding notifications to *that* email, as opposed to a different address linked with their eBay or Amazon account.

Why doesn’t the scammer simply send their fake email through the marketplace messaging system? Some marketplaces simply won’t allow external links to be included in messages sent within their system. Some marketplaces won’t allow email addresses to be mentioned. They will monitor for anything that looks like evidence that trading is taking place (or being encouraged to take place) outside their marketplace, and will move to ban both Buyer and Seller.

So the scammer needs the Seller’s PayPal email. They may not baldly ask for it. To avoid suspicion, they may camouflage their intent in a request for other information. Often, more photographs or requested, or a separate invoice. As many marketplaces won’t allow mention of email addresses in their messaging systems, the scammer has to be a bit creative here. For example, if the Seller helpfully sends an invoice “for their records”, it may include an email and phone number. The Scammer will make contact using these details, and request the PayPal email to complete payment. Once they have the email, that false notification is sent.

Best Defense

guard-dog

 

Clearly, the best defense against this scam is for Sellers to check their PayPal accounts to verify that they have been paid. Don’t click on links within emails to go to PayPal, remember the cautionary note about phishing.

Open a browser page and type www.PayPal.com.

 

 

Variant – “Payment on Hold”

To counteract sensible Sellers checking their PayPal account, scammers will use some additional sophistication. Another fake email will arrive, purporting to be from the Marketplace. This one states that the Marketplace (e.g. eBay) are holding the funds until the Seller emails the tracking number of the item to the Buyer. Of course, the provide the tracking number, the Seller has to go ahead and ship the item. Ingenious.

This is completely fake. EBay or Amazon never hold funds in this way.

This email takes several forms. Here is one example from 2016. This is a double-hander, the Seller got two emails.
Email #1 from Fake PayPal: “You have funds of X pending to be credited to your account. When we receive the Tracking Number from the buyer via email, your account will be credited immediately.
Email #2 from Scam Buyer: “Please send me the Tracking Number, so I can send to PayPal, thank you.

 

When Sellers are Culpable

Sometimes, the Seller is not a faultless victim. When contacted by the scammer, the Seller may decide to keep the transaction outside the Online Marketplace to avoid fees. That is one reason to voluntarily give up one’s PayPal email. Seller Beware!

No Cash! Or “worse things happen at sea, you know”

Obviously, as this scam depends on payment via PayPal, if the Seller’s preferred method is Cash on Delivery then the Scammer must convince the Seller to switch. This usually involves some cock-and-bull story that puts the scammer all-at-sea. Here’s a few examples reported by Sellers:

sailor-duck

“I work offshore as an instructor on an oil rig.”

“I am an oceanographer.”

“I’m a Marine contractor at sea.”

 

But not every scammer is a jaunty sailor. This poor dude had other problems: “sadly I can’t come personally to collect due to my hearing loss and I’m just recovering from heart surgery so I’m home-bound.” In his next email, the unfortunate fellow’s leg had fallen off.

 
Launching Soon: “eBay Selling: Avoid Scams and Fraud when Selling Goods Online”

Fake PayPal Notification – how not to label this scam

Surfing the web for info on eBay fraud, I stopped to read an article posted by the Daily Mirror, the online site of a British tabloid newspaper.
It describes four scams perpetrated on eBay sellers, the first one headlined as “Send it to Nigeria“.

So we’ve got a catchy easy-to-remember title for this class of scam. I read on to learn the details. I scour the two paragraphs, which are succinct and clear, and wonder what I’m missing.

fail

 

“The buyer says they’ve paid more so you can send it to another country (several sellers said Nigeria). They also ask for your Paypal email.”

“Sellers said they were contacted by ‘Paypal’ asking for more personal financial details.”

 

So the scammer asks me to ship the item I’m selling to Nigeria. Got it, I’m taking notes here, Daily Mirror. This is proper investigative journalism. Wait, what? Where is the fraud? I’ get my money, I send the goods. Happy days in Liverpool and Lagos.
Way to go, Daily Mirror, for missing the bleedin’ obvious. The problem here is the bogus contact from PayPal. I mean, the journalist actually puts Paypal in inverted commas, so she does know what’s she’s talking about. Presumably the editor didn’t bother reading as far as the second paragraph, got stuck in those parentheses (several sellers said Nigeria),  and then chopped the rest of the scenario that would explain the scam. To avoid actually clarifying the situation, a link is given to a ten-year-old Ebay forum thread. Investigative journalism, it rocks.

Look, 100% of of scammers committing this fraud may be from Nigeria, but that doesn’t tell us anything. This scam is all about being sent emails purporting to be from Paypal.

Here are the details of the actual fraud.
1. Seller receives an email from the scammer stating intent to buy the item.
2. Seller receives an email that looks like it comes from PayPal, and looks like a notification that funds have been transferred from the scammer.
3. Seller posts item.
4. Seller realizes that no funds have been transferred to their PayPal account, and that the email was spoofed.

Spoofed? How does that work? How do we avoid this scam? More posts on this to come…