Sounds Like Weird
22 February 2015
Recently, on the #notchilled hashtag, we heard about Ellora’s Cave’s IT system in the days of yore. Then Ellora’s Cave sent an internal email on the subject. I have some commentary after that.
First, the tweets that started it all:
@pubnt @Format_Me You’d think they would have learned. Back in ‘03-04 then COO and my husband tried 2rantically save only copy #notchilled
— BJ (@JetGibbs) February 14, 2015
@pubnt @Format_Me of publishing AND royalty data on sick laptop. I was there, you weren’t, Pubby #notchilled
— BJ (@JetGibbs) February 14, 2015
@pubnt @Format_Me At the time, husband strongly suggested offsite redundant backups (IT security professional). #notchilled
— BJ (@JetGibbs) February 14, 2015
@pubnt @Format_Me Funny, now that I recall JB and PM were in the bar. Partying. #notchilled
— BJ (@JetGibbs) February 14, 2015
Note: this happened in the 2003-2004 time period. Okay, now that we’ve got that cleared up, let’s read the rest of BJ’s tweets on this topic…then Ellora’s Cave has an official response.
@Format_Me DH actually offered his services but the price was…um not right.
— BJ (@JetGibbs) February 14, 2015
@Format_Me @pubnt Any company living and dying by their data should have redundant offsite backups.
— BJ (@JetGibbs) February 15, 2015
Spoke with hubby to make sure I was correct He said that yes, everything regarding the company infrastructure was on that one dying laptop.
— BJ (@JetGibbs) February 15, 2015
On 1 EC company PC was only copy of manuscripts, financial dox, royalties, tax dox, payroll. The works. #notchilled
— BJ (@JetGibbs) February 15, 2015
@JetGibbs Wait. EC had their entire company files on a LAPTOP????? What idiot made THAT decision? #notchilled
— tejas (@tejasjulia) February 15, 2015
And there were NO backups. Just that one EC laptop with keys to kingdom @tejasjulia #notchilled
— BJ (@JetGibbs) February 15, 2015
@ShoshannaEvers Its for real. I worked as an editor for them for 12 yrs. This was at RT in Kansas City (03? 04?)
— BJ (@JetGibbs) February 15, 2015
@ShoshannaEvers Crissy had been in IT and she and my husband connected on that level, so she brought him in when the laptop was sick after
— BJ (@JetGibbs) February 15, 2015
@ShoshannaEvers one of the events–Jen Martin (then an editor) Crissy, I think there were a couple other folks in the hotel room.
— BJ (@JetGibbs) February 15, 2015
@julainestone The RV was about the same time as the single laptop. Maybe the year later? There was much rejoicing and fanfare.
— BJ (@JetGibbs) February 15, 2015
@ShoshannaEvers And yes, they did have a HQ, but this was at an RT convention. They had several (the warehouse in Akron, and previous
— BJ (@JetGibbs) February 15, 2015
@ShoshannaEvers a business complex in maybe Kent or thereabouts. They ended up splitting editorial to a complex across the st.
— BJ (@JetGibbs) February 15, 2015
@ShoshannaEvers I hope so :/ I never worked in office (freelancer) and only went to one RCon so I haven’t any idea how things had been run
— BJ (@JetGibbs) February 15, 2015
@ShoshannaEvers in the last however many years 😉 But I did consider once (circa mid 00s) working in office and spent a week out there
— BJ (@JetGibbs) February 15, 2015
So, to recap what I understand as the essential elements from the above:
So Ellora’s Cave’s CEO, Patty Marks, sends a puff piece to their biz group. Which, by the way, happens to be hosted on yahoogroups, exactly where I’d (not) expect a company with a competent IT infrastructure to host critical infrastructure mail groups….
Letter follows.
Sent: Monday, February 16, 2015 2:36 PM
Subject: [ec_biz] Gossip regarding our systems
I don’t like to address gossip, because it lends to the possibility that someone may find it credible, but I think this is important.Apparently there is someone out there saying that we don’t back up your information and that we run our systems on single computers without backup.This so ludicrous that it shouldn’t need addressed, but just in case…
All of our data, including but not limited to Financial, Manuscripts, Graphics, Spreadsheets, programs and any resources used in the daily business of the company past or present is currently stored in a multi-server network.Each server is raided in the event of a hard drive failure and run we run redundant power supplies as an extra precaution.All data stored on the network is then backed up to another location using automatic backup software.The entire room is on its own electrical panel with commercial grade surge protection and battery backup.Our server network and all computers that access it are protected by the latest in antivirus and firewall technology.We have and still do employ a full-time IT department since 2005 – not to mention three outside consulting and hosting companies, one for the website, one for our computers and servers and one for our accounting programs.
Before that, we were a company consisting of six people or less and had no server network at that time.IT services were subcontracted on an as needed basis.Computers were backed up individually to external storage devices and no one computer contained all of the company information – at least not since I have been with Ellora’s Cave.As the company has progressed so has our hardware and software that we use every day to run things.
As for the confusion in the data loss with the Amazon cloud crash, there was no royalty data lost.The information erased were certain formulas that were built into the back end of the old EC site.We had the consultants who designed the original formulas fix the broken code – using the backups – and install it on our servers as a standalone program.All financial spreadsheets, imported or exported, royalty programs hosted offsite or onsite, have always been stored on our server network as well.Again we run like seven server environments for redundancy.
Our email system and our website all run on multiple cloud based servers for backup redundancy.Our own network has its own backup redundancy as described above.All user passwords and master passwords have to meet strict password requirements and are changed on a regular basis.All internal users require a domain credentials to access the files they have been give permissions to view and remote users need their domain credentials as well as firewall credentials to gain access to the internal network.No one person has access to the entire system except the IT department.All other permissions are locked down by department and department level basis.
AND – on top of it all – we carry business insurance in case of hardware or software failure and data loss.
Can I say with certainty that our systems are absolutely impenetrable?Well, considering people have hacked banks, giant retailers, SONY and managed to crash the Amazon cloud, I would guess not.But to imply that all of our information is on one “dying laptop” is laughable and simply untrue.
I’ve lost data, and it’s not pretty. In fact, I’ve been considering writing two separate pieces about preventing data loss: one about several simple things you can do to ensure you don’t lose your data on your hard drive, and the other would be ways to ensure you have a full backup of a WordPress site. The latter involves some harder problems, though.
I have what I consider a paranoid level of backups. The other day, I had to wipe my server (that no longer hosts deirdre.net but still hosts several other domains I run).
I bought another domain the other night and was wondering where I was going to put it—my VPS (virtual private server) was out of space. Then I remembered I’d paid for more space, so I should just use that. Except that meant re-partitioning the drive, which meant wiping it, and then I decided I wanted to switch Linux distributions while I was at it. So…been at that for a few days.
I use the utility rsync to push my sites to my VPS. This is an old habit of mine.
Each site is kept in its own git repository. For those unfamiliar with git, it manages source code control. So, essentially, I can get back any revision at any point. Git leaves an invisible directory, .git, at the top of the repository.
Unfortunately, when I pushed one site up, I forgot to exclude the .git directory, and I canceled the transfer in the middle. Remember, I was space-constrained and git repositories aren’t particularly small.
Then I forgot to remove the partial .git directory.
So I’m in a hurry to back up my domains before wiping my VPS and resizing the partition and reinstalling and yada yada yada, and what I meant to do was to copy the images/ directory from the server back to my own hard drive, so I didn’t lose any photos. As Jenny Trout did. Disclosure: I have made the exact same mistake Jenny did, and that’s why chair-in-the-sky.com doesn’t have any of its old posts. I’ve removed them until I can dig up the photos. Sigh. Like Jenny, I lost them in a site transfer. Of all the steps in moving WordPress installs across sites, the media library is the most fragile part.
Rewinding a bit: I’m in a hurry. Instead of that one directory, I rsync back to my hard drive the whole domain, including the partial .git directory.
Clobbering my repository.
Sad panda.
But!
I have always pushed my git repository to another site! It’s all good, because my backup’s only a few minutes old. I can re-init a git repository and pull from the remote site.
However, the quicker option? I restored a thirty-minutes-ago backup of that directory with Apple’s Time Machine. Then I correctly pull the uploads directory only and add the new contents to my git repository. And push it upstream.
::phew::
To many of you, what I just wrote sounds completely unintelligible, but I do have a point in writing it (apart from personal embarrassment).
I’m a one-person shop, and this is the skill and expertise I have to ensure my own data doesn’t get lost, even when I screw up when I do something I’m too tired and/or rushed to do correctly.
Ellora’s Cave would need that much more skill on the ground, and their staff list doesn’t reflect that.
I suppose they can just pay for all that IT infrastructure out of the bank account with $15 million in it, right?
And @pubnt wonders why Dear Author and Jane Litte want to subpoena her.
The Ellora’s Cave Author Exodus Support Thread helps support EC authors who’ve spoken out by linking to their sites and books.