Joe Winett And the deletes are done. 5,443,938 records. I guess one delete failed and it failed between midnight and this morning beccause I did the math and it was supposed to be ..7.. hmm I’m not going to freak out, I don’t think.
[8:33:37 AM] Joe Winett: It was doing upwards of 1000 deletes per second at one point. I’m guessing that the last few imports were changed back to only loading 1 image… OR perhaps I deleted the queues at some point and forgot?
[8:40:38 AM] Joe Winett: There are about 12 million images in the CDN.
[8:42:12 AM] Joe Winett: 318gb or $31.80/mo (Rackspace lowered their price from 0.18 to 0.10. for some reason I thought it was 0.12 the other day)
[8:48:42 AM] Joe Winett: Six slaves overnight were a total of $0.18/hr. They’re gone now. We’re left with the two 2gb instances of $0.12/hr each. I’m going to go ahead and resize the second down to as small as it should go to contain the database.
[8:50:29 AM] Joe Winett: What’s cool about that the load balancer instance ($10/mo) is that while the second server couldn’t be doing mongo because I thought it was stupid to replicate all the deleteing, I also turned off the web service and then went in and made it inelegible for load balancing.. so all web has been served from one of the servers… The public IP address for *** is the IP address of the load balancer..
[8:52:28 AM] Joe Winett: Compacting the mongo database from WAY big to reasonable will cause… OH WAIT… I think I can make this all happen with it going down. Replication again… HAHAHA yay mongo
[8:54:40 AM] Joe Winett: Replication is not good with only two members because if one goes down, the single remaining member refuses to be the President (weenie)… Actually, it’s just computer science… the last remaining member doesn’t know he’s not been marooned on a desert island with no internet, so it decides the best thing to do is not accept any “primary only” requests.
[8:55:00 AM] Joe Winett: A primary only request would be anything that has to be replicated. Like a write or a delete.
[8:58:01 AM] Joe Winett: For best performance in our case, we need three servers of moderate size. They can elect a leader themselves (I’ll put weight on the first). But, we’ll span reads to any of the three servers. So, there won’t be global locking always blocking all reads… There’s locking when the replicated writes get to the slave server (a matter of milliseconds) but some reads from the web servers will be available from the other servers depending on the timing of it all.. which is unpredictable.. Requests from the load balancer will go to the server with the lowest response time… which is unpredictable. I might just make that random.
[9:00:48 AM] Joe Winett: And the queue processes will run on all the servers. We had 9 processes each, but mongo wants ram, so it will probably be just 6 processes each. And then I’d thought I’d manually start slaves again when we do an import to see how much traffic Rackspace can handle by day and night… and then we can have the master queue process automagically turn on and off slaves as needed (perhaps based on the uploading times to Rackspace, or download times from our provider one day). The queue.php scripts can decide who is master by asking the mongo on that machine who won the election. Eagles fly with Eagles.
[9:16:51 AM] Joe Winett: The database should fit comfortably in 8bg of space. So, we have lots of options. We were paying $0.24/hr for two 2048 so we could get 80gb of space to store the waste. But we need three instances.. 256mb of RAM just shouldn’t be allowed (actually, it would be too small, but I wanted to make the joke). It might work alright in 512mb for each instance, a total of $0.09/hr. If we need to 1gb instances, then it would be a total of $0.18/hr. Obviously, the thing to do is try to make it with 512mb instances and not run mediaq slaves on those servers… have the media master-in-chief start other 512mb instances to do that work, only when it’s there. GOOD PLAN. Glad we spent this time together.
[9:48:19 AM] Joe Winett: Who called this meeting?
[9:48:41 AM] Joe Winett: I like Skype. It’s just like talking to yourself in your cubicle, but there’s evidence you were there.
[9:49:37 AM] Joe Winett: I just dripped coffee all over the database statistics. That’s what you get, sometimes, when you write things down… coffee stains or dogs eating them.
[10:33:51 AM] John Q Programmer: I didn’t notice that my computer hadn’t auto-prompted me to login and it looks like I missed the whole discussion. The numbers look god though. Not my opinion deserves much weight as I have no server skills and am easily mesmerized by paragraphs with lots of numbers.
[10:39:33 AM] John Q Visionary: excellent news!
[10:40:10 AM] John Q Visionary: so now John can add the delete into his code and we can run and import and test it out?
[11:09:20 AM] Joe Winett: Yes, but
[11:09:47 AM] Joe Winett: Please don’t run an import until tonight when I have everything resized.
[11:14:58 AM] Joe Winett: John — I’m mentioning all this stuff because cloud is definitely the way to try new things out and previously unimaginable project scales, with burst capacity, can be attained for little to no money. So, I guess I’m mentioning it because the sooner the seed starts growing in your head.. haha I’m pretty sure that a large scale mongo deployment, unless we do sharding with small servers, might be easier done and less expensive on our own hardware just colocated in a good place… Oklahoma City isn’t bad for that, but we’re going to want to probably replicate to a backup server… A single massive box, with enough RAM to fit all the indexes, and lots of cores, should be able to crank out with low latency. There are no database join clauses, so we have to work out the way attributes will function using our noggins and then just writing code to do it… probably Javascript code that runs on MongoDB itself. BUT, what we have seems to work really really well at the moment.
[11:38:26 AM] Joe Winett: Oh wait, I’m not going to have to actuall compact the database… just replicate it to the slave, then make the slave the master. Did I write that before? Oh, wait, I don’t read my own writing… who has the time?
[12:35:31 PM] John Q. Programmer: I appreciate you filling me in on how we’re using cloud power. I’m just not yet at a point to make intelligent comments about it. But staying radio silent seems a bit rude of me. End result, silly comments about the fact that I can only make silly commets.
[12:36:42 PM] Joe Winett: It’s just a matter of messing with it… It’s just like using a server we set up on our own box, or on your own emulator, except they have a web interface to turn up a new server of whatever flavor you want… And you can image it and then turn it back off… and you pay for the hours you used… (and for $0.10/gb of storage for the image per month).
[12:39:37 PM] John: Right. I understand the theory. But I’m still working on the sepcifics. Next time I have a few moments maybe I’ll go read their documentation until things click. My past experience has all been single server with simple clients so the sheer scale of this project still leaves me a bit unsettled. Growing pains and learning curves.
[12:41:07 PM] Joe Winett: Well, get the password and just start a server and log into it. You can always turn it off. JUST DO NOT DELETE our live servers. It’s amazingly easy to click DELETE and then YES without thinking. They need to have people type in something… or be able to put protection on them. I know Amazon has protection. I’ll look.
[12:41:49 PM] Joe Winett: I did the resize on the second down to 512mb and it all fits nicely on there. Putting mongo into the group to start a replication.
[12:42:21 PM] Joe Winett: It’s getting hot in the house right now. I’m going to leave around 2 or 3 to woo a woman in the comfort of air conditioning, but this should all be cool before I leave.
[12:42:54 PM] Joe Winett: (I have air conditioning, but this 1972 window unit is out of its league at the moment. The Big Cooker is mighty.)
[12:43:01 PM] Joe Winett: haha mighty . Pun intended.
[12:43:36 PM] Joe Winett: And I’m living in a tv dinner at the moment. Wow, I am way going to appreciate apartment living one day.
[12:43:54 PM | Edited 12:44:08 PM] Joe Winett: But I’ll miss this patch of Earth. Good trees.
[12:44:21 PM] Joe Winett: WOW!?!? You can edit a message on Skype. That Other Guy was right, this is cool.
[12:44:49 PM | Edited 12:45:08 PM] John: You mean edit an existing message? I just changed history!
[12:45:23 PM] John: That was vaguely unsettling.
[12:56:31 PM] That Other Guy: Too bad it only lets you change your last message. 🙂
[12:57:07 PM] John: Too bad. I was planning on retroactivly removing all my bad ideas and spelling mistakes.
[12:57:11 PM] Joe Winett: I have a better groupware idea in my head, but I’m not going to spend any time on it.
[12:57:51 PM] Joe Winett: I’m sure they’re out there… and yes, like a threaded forum, but if it was laid out like developers use them and looked a bunch cooler, and was integrated with Google Apps, then it could make lots of $
[12:58:13 PM] Joe Winett: …with video conferencing… That Other Guy is going to say it’s out there.. .That happens to me a lot.
[12:58:43 PM] Joe Winett: But not full time video — PUSH TO TALK video. that is attached with the message
[12:59:39 PM] Joe Winett: and push to talk audo, like Nextel phones. With a USB CB microphone to plug into your computer… No geeky headsets. That microphone idea would make millions, I think.
[12:59:45 PM] Joe Winett: BREAKER BREAKER GOOD BUDDY
[1:00:13 PM] Joe Winett: Just put the speaker in the microphone, like a HAM radio or police talky mic.
[1:00:31 PM] Joe Winett: That needs an engineering team (or Steve)
[1:00:56 PM] Joe Winett: And Nixon to go to China to negotiate to have them made, but he’s dead… bother.
[1:01:23 PM] That Other Guy: lol … and I don’t know if that’s out there or not. WIndows communicator might have something like it.
[1:01:28 PM] John: The problem with selling things to developers is that as a group they actually enjoy building their own free versions of comercial tools. Why spend $100 on something when you can get it for the low low cost of five years worth of your free time?
[1:01:40 PM] That Other Guy: ROFL Nice one John
[1:01:50 PM] Joe Winett: Exactly. I used to be that guy.
[1:02:04 PM] Joe Winett: I also wore the band’s t-shirt to their next concert. I was that guy.
[1:02:55 PM] That Other Guy: I made my own t-shirt >.<
[1:04:13 PM] Joe Winett: AWESOME!!! You win, but that doesn’t make you “that guy” that makes you “the guy”
[1:05:16 PM] Joe Winett: I want to make t-shirts so badly I can just taste the dyes. Again, why buy a t-shirt when you can spend $100k on equipment to make your t-shirt
[1:06:06 PM] Joe Winett: Same thing with printing my art. It’s just not worth it to have someone else print it… It’s the fun of the printing and mounting or whatever that makes it fun… And, again, like $3000 worth of printer and supplies… and a place to put it.
[1:06:20 PM] Joe Winett: So, I need like $1.3 million dollars and a helicopter.
[1:06:49 PM] That Other Guy: Have you considered a painball gun? They’re somewhat cheaper and a lot more fun.
[1:07:02 PM] That Other Guy: Plus, you can mount them on cars fairly easily.
[1:07:17 PM] Joe Winett: And I want to suspend a light array from the helicopter and drive around OKC playing DUN DUN DUN DUN DUUUUUUN from Close Encounters…. until they make me stop it.
[1:07:26 PM] Joe Winett: hahaha
[1:07:52 PM] Joe Winett: I’d love to see that ticket for endangering public safety with an aimable paint ball gun turret mounted on your car.
[1:08:31 PM] Joe Winett: And I want to make a computer case for gamers that has little air shocks. Normally it’s a low rider, but when someone gets in your business it can puff your case up to show your attitude.
[1:08:55 PM] That Other Guy: LOL Nerd-rage indeed.
[1:10:00 PM] Joe Winett: So the case need wheels that look like tires… and you can buy accessory special wheels or wheel covers… This one is ACTUALLY solid titanium. And you’d want to make at least one (fake) model of gold and platinum. SOMEONE would order a set… and you have a fewler make them… Ok, they wouldn’t be solid enough… so, they’re steel with actual gold plating
[1:10:16 PM] Joe Winett: (Or you can order the actual gold painted set for just $19.95)
[1:10:39 PM] Joe Winett: Well that will make millions too. Need an engineering team…
[1:11:42 PM] Joe Winett: And AMOLED wrapped commercial and residential fixtures like colums… Bars would pay for those bad boys… They could show anything… like bubbles… They could react to people being near them using sensors… and show fish swimming “inside” the columns. MILLIONS … IT WOULD MAKE MILLIONS
[1:12:20 PM] Joe Winett: But we’d have to make a scale model unless we already have the 1.3 million and/or the helicopter.
[1:12:33 PM] Joe Winett: The helicopter can make money by giving rides to casinos.
[1:12:40 PM] Joe Winett: ON A STREAK?
[1:13:57 PM] Joe Winett: (kidding… never going to make money from Casinos, I’m pretty sure…) I turned down a job in Tulsa with BIG GAMING DUDES, one of the largest producers of Bingo based fake slot machines for Indian casinos… It was for a programmer and support tech… They were shifty as all heck in the interview. Kept looking at each other after every answer I gave. But it was lots of money and free catered lunch every day. I didn’t sell out.
[1:14:39 PM] Joe Winett: And I was homeless, so that was hard. Took a bus to the dry cleaners… and then BACK the day before to have my suit cleaned… and then to the interview and walked 1/2 mile… and showed up early enough to unsweat myself.
[1:15:16 PM] John: I didn’t know bingo based gambling was such a hot ticket money magnet
[1:18:25 PM] Joe Winett: They were the first kind of slot machines allowed in casinos because they actually play bingo… not slots. But people don’t understand the odds and how they work — you’re playing against everyone else in the casino… I suppose it doesn’t matter, but if someone next to you just won a lot of money, you’re NOT going to win (not possible) to win the same card in that game… But people don’t know to look for when there is a new game… The numbers are already chosen and everyone plays with the same numbers until the end of the game… You’re just buying another card for that same game… The casino limits their loss in each game by only throwing in a certain number of the big winning cards, but they’re off the hook when a “game ending” card is played. I suppose it doesn’t matter because in a large casino they spread everyone out so you can’t hear who’s winning or whether you should continue in the game. SO, they’ve cheated people out of the advantage they have in the real bingo hall, to run up and buy another card and then quickly blot down the numbers to see if they can still win.
[1:19:17 PM] John: Clever.
[1:19:53 PM] Joe Winett: But the real bingo cards aren’t that specific… so the casino has less of an advantage there… But on a real slot machine, you have the same chance of winning with every spin.. which is really really low, but at least it’s the same as the last spin. Sometimes on bingo you really have no chance of winning. Plus, they’re so shifty… I don’t know that they don’t allow their clients to adjust the cards… I can’t believe they’d ever get away with that, but they might.
[1:20:12 PM] Joe Winett: Best not to gamble on anything… especially machines.
[1:21:34 PM] Joe Winett: And BGD makes money by leasing the machines to the casinos along with a 24×7 support agreement. That was going to be part of my job, to be on call for one week every third week (we would trade off). and if the THAT TRIBE called, then I’d have to drive out there (didn’t have a car) and take care of the problem if it couldn’t be done remotely. They told me that when the TRIBE’S machines are down when they’re busy that they’re losing [lots of zeros] per hour, so they get a little hot.
[1:21:52 PM] Joe Winett: Said they might be a little rude on the phone and asked whether I could take it.
[1:22:28 PM] John: It’s no fun being blamed for entropy.
[1:24:14 PM] Joe Winett: hehe, yeh. They said the problems were almost always created by the clients changing the ethernet network. They install their own network and then one port in a switch to go back to the management box(es)… But sometimes the casino would install a new bank of something else or need a port and then try to mess with it to save wiring expenses, but the bingo machines each use a lot of bandwidth to keep in contact with a central server that is “pulling” the balls out of the basket.
[1:24:44 PM] Joe Winett: and reporting back to management and handling money with the casino’s banking unit — there’s a standard protocol for all that… because all machine share the banking system for the tickets.
[1:26:02 PM] Joe Winett: And that’s pretty good security from keeping a slot machine manufacturer from just printing tickets… The banking system keeps track of the property IDs to look for cheats… But a programmer could spread it around. WHICH IS WHY, in Tulsa, they can’t change any of that code… It’s not even on disk.. It’s on ROM… it’s written somewhere else.
[1:27:02 PM] John: That’s pretty impressive
[1:27:54 PM] Joe Winett: They catch and fire a lot of cashiers for stealing. But their loss is limited there by how much cash they handle in a day. They’ll cat the thief sometime in the next few hours…
[1:28:54 PM] Joe Winett: And a lot of places have gone to automatic money dispensers. So, the cashier theft is limited to money for change… like a drawer in a store. If they want to steal $100 bills, then they have to steal them from the people who just won and that’s not happening
[1:29:50 PM] Joe Winett: But there are some who still have access to wads and wads of money at each window. Scary for them, I’m sure… But EXCELLENT for the companies making those dispensor machines.
[1:30:43 PM] Joe Winett: It’s all fascinating, but I’m never going to mess with it. And I wouldn’t mess with banking except to be a better bank, but I think some are already better banks (or credit unions).
[1:31:03 PM] Joe Winett: Maybe to make micro loans for businesses… Up to $1000 if the story is pretty good.
[1:31:50 PM] John: Have you looked at Kickstarter? It’s an interesting take on micro-loan / casual venture capital / charity.
[1:32:35 PM] Joe Winett: And you make it where they have to have 25 adult cosigners. Then you make the loan, no questions asked, and no collection procedure… But those 25 people cannot sign on another until it’s repaid.
[1:33:03 PM] Joe Winett: And you make it easy to sign… Just come in and show your ID. No actual signature or promise required.
[1:33:55 PM] Joe Winett: So 25 people could make a pact to steal $40 each with no intension to repay, but you’ve got to figure they need the $40.
[1:34:51 PM] Joe Winett: But if I had another busines, like an online business, that was making boatloads of cash, then it would be worth it. And you put the numbers on the bank’s website… $1,000,000 loaned, $800,000 paid back in this community…
[1:34:58 PM] Joe Winett: Do that for 25 years and then run for President. 🙂
[1:36:27 PM] John: The leftists would claim you were cruel for expecting a 80% payback rate and the rightists would claim you were crazy for accepting a 20% loss.
[1:36:38 PM] Joe Winett: Most payday lenders have to charge up to $25/hundred for a short loan to make up for that loss.
[1:36:55 PM] John: And the libertarians would refuse to vote for you due to not being Ron Paul. Not much you can do about that one though.
[1:37:30 PM] Joe Winett: But no, it’s basically charity in some cases, but they have to get 25 people to agree that it’s worth it to them for this person to get that money. I bet you’d see the community paying back some on the loans… Come in with your ID and make a payment… $5 whatever on that loan… Maybe anonymously.
[1:37:55 PM] Joe Winett: Well, it doesn’t matter if I actually make it… I will, but it doesn’t matter. 🙂
[1:38:19 PM] Joe Winett: There’s no point in having a bunch of money in the bank… so I can buy Rhode Island?!?
[1:38:52 PM] Joe Winett: I bet you’d see people making payments across ALL the loans in a community…
[1:39:03 PM | Edited 1:39:16 PM] John: Actually charity loans would probably be better for people’s character than pure charity. Give someone a sense of obligation and they have that much more drive to pay back the community instead of asking for more.
[1:39:20 PM] Joe Winett: Exactly… NO QUESTIONS ASKED LOANS
[1:40:51 PM] Joe Winett: Kickstarter would love it, but you really need to do it at physical locations. Perhaps a kiosk that could be put into a business. But the business would have to hand over the cash, or write a check… then they’d be on the hook for making sure the ID was real. It could be worked out.
[1:44:34 PM] Joe Winett: I’m going to copy a bunch of this — my ideas and some of the information on the cloud stuff (without too many particulars) to my blog unless there is an objection. I thought I would refude your names just first names. I thought from the information about cloud computing and on. If someone does some of that stuff, then cool… Do all you want, we’ll make more.
[1:45:01 PM] Joe Winett: Perhaps each of you should actually agree. I won’t do it unless you all agree.
[1:45:28 PM] Joe Winett: That’s how to become President: “Perhaps each of you should vote for me. I won’t do it unless enough of you vote for me.”
[1:46:23 PM] Joe Winett: 100% percent of the votes would be best, but if enough go with the Energy, then I’m willing and able to serve just 4 really awesome years, then I’m going to go eat my ice cream by the lake.
[1:46:53 PM] John: I’m close to paranoid and not particularly fond of being referred to on the internet (silly desire in the information age, huh). But I can’t disagree if you want to repost what is essentially your own lecture on cloud computing.
[1:47:28 PM] Joe Winett: I can take your name out completely. Or!! You can have an internet fake nam e. Mine is Jerry Wallace, so don’t pick that one.
[1:47:53 PM] Joe Winett: And yes, that’s a problem because your name will come up with my name… and I’ve talked a lot on there.
[1:48:05 PM] Joe Winett: So definitely, no last names.
[1:49:30 PM] Joe Winett: But I’ve written that my fake name is Jerry Wallace, specifically so people can find other things I’ve written. So, I guess I need a new fake name.
[1:49:41 PM] John: You could always just refer to conversation participants other than yourself by numbers or letters. I believe a lot of writers do that when they want to focus on the conversation and not the random participants.
[1:50:05 PM] Joe Winett: Like when writing in a psychology book about studies… or in market research. YEH.
[1:50:47 PM] Joe Winett: Plus that cannot be associated back very easily. I mean if people want to find out where I’ve worked I don’t think that will be hard. I think Linked In says where I work and lightning has yet to strike your Global HQ.
[1:50:56 PM] John: My sister actualyl loaned me a book with excerpts from Andy Warhall. He referred to all his friends as B and you never knew which one.
[1:51:01 PM] Joe Winett: Oh, Facebook does too.
[1:51:11 PM] Joe Winett: hahah that is funny
[1:51:19 PM] Scott Cornaby: Anyways apparently I have to go look at offices with Steve RIGHT NOW so I need to log off. Bye.
[1:51:55 PM] Joe Winett: Have fun!! I think separate little offices would be best because we don’t talk much when we’re there anyway, but then we’re all there for yelling across the hall and throwing paper wads.
[1:52:05 PM] John B: Noted
[1:52:22 PM] Joe Winett: Nerf guns.
[1:52:34 PM] Joe Winett: Squirt guns filled with dielectric oils.
[1:53:09 PM] Joe Winett: Oh, just thought of a new adult product to be sold in the more discrete retail outlets. I’m not above making sex toys.
[1:53:20 PM] Joe Winett: With my picture on the package.
[1:53:32 PM] Joe Winett: “VoteJoe50.org” Adult Novelties!
[1:53:55 PM] Joe Winett: So see, people can dismiss it all as just a promotional gag until it’s TOO LATE TO STOP ME.
[1:54:03 PM] Joe Winett: MUAAWWWAAHAHAHAHAAAAA
[11:43:56 PM] Joe Winett: Ok, FREAKY. I turned back on the replica set setting and it just started replicating to the now empty #2, but I thought I’d removed the configuration. Well, gift horse, mouth.
Leave a Reply