Fun with Tintri Part 2

In my last post I talked a bit about the hardware and getting set up with our Tintri VMstore T540 system which went very smoothly. I had thought about writing my next entry about going through the initial system setup and configuration but to be honest it's so easy I didn't think it was necessary. Part of me wanted to start throwing some workloads on this thing and see how they performed but I needed to collect at least some performance data before I started moving my existing stuff around. I've never really done any serious I/O testing before and I wasn't entirely sure where to begin, but eventually I stumbled upon a fling provided by VMware called the I/O Analyzer and this thing is a fantastic tool. You download and deploy a small OVA template that creates a lightweight Linux VM. The cool thing is that you can deploy multiple copies of the appliance and control them all from a single instance. There's a great video by Chris Wahl that shows you how to quickly get up and running with this appliance and I highly recommend anyone looking to learn how to use it check it out.

So after watching the video that Chris created I dived right in and got started running a number of different tests against my current storage system and the Tintri T540. There's a few notes/caveats for my setup here that I did want to share:

• The Tintri system did not have any active workloads other than the test workers on it at the time of testing. My current storage was running an active production workload but it has enough additional capacity to allow for the testing to run concurrently.
• The current storage system I'm using does have some flash onboard but it is only servicing read operations whereas the Tintri's flash is serving all of the I/O operations.
• Both my current storage and the Tintri are serving storage to my VMware environment via NFS.
• Testing Equipment consisted of 3 Cisco UCS B200 M2 blades which have Dual 6-core Intel Xeon processors and 96 GB of RAM. All of my networking is running at 10 Gig.

My goals with this testing were twofold: One, determine what certain workloads would deliver in terms of throughput on each system. Two, see just how far I could push the Tintri to get an idea of just how much performance it can deliver. Without any further delay, let's take a look at some of the results:

VMware I/O Analyzer Tests

Test 1 - Single Worker Process OLTP 4k - 5 Minute Duration

Storage	Workload	Total IOPS	Read IOPS	Write IOPS	MBPS	Read MBPS	Write MBPS
Current Storage	4k 70% Read 100% Random	8355.75	5850.52	2505.23	32.64	22.85	9.79
Tintri	4k 70% Read 100% Random	20947.37	14664.28	6283.08	81.83	57.28	24.54

So this first test was supposed to mimic the I/O patterns of a typical OLTP database workload. You can see that while both systems handled it relatively well, the Tintri pushed more than double the IOPS of my current array in the same 5 minute duration. My company does run a decent sized OLTP workload so this test was of significant interest to me to get an idea of how one of my current systems might operate on a Tintri system vs. what I can deliver today.

Test 2 - Single Worker Process Exchange 2k7 - 5 Minute Duration

Storage	Workload	Total IOPS	Read IOPS	Write IOPS	MBPS	Read MBPS	Write MBPS
Current Storage	8k 55% Read 80% Random	3338.28	1833.16	1505.12	26.08	14.32	11.76
Tintri	8k 55% Read 80% Random	12837.28	7057.01	5780.27	100.29	55.13	45.16

In this test I used an Exchange I/O pattern simulation which was of interest as I run Exchange in the environment as well, but we really don't have any major performance issues with it. Even so, it was good to see how this workload would be handled on each system. You can see that the Tintri didn't push as many IOPS as it did with the OLTP simulation but it makes sense given that the I/O was more evenly split between read/write operations and the block size went up to 8k. Regardless of that it still did about 4 times the IOPS as my current storage so that was great to see.

Test 3 - Multiple Workers w/ Varying Workloads - 5 Minute Duration

Storage	Workload	Total IOPS	Read IOPS	Write IOPS	MBPS	Read MBPS	Write MBPS
Current Worker 1	4k 70% Read 100% Random	1509.71	1057.44	452.27	5.9	4.13	1.77
Current Worker 2	8k 55% Read 80% Random	5005.66	2749.87	2255.79	39.11	21.48	17.62
Current Worker 3	8k 95% Read 75% Random	3225.18	3062.74	162.44	25.2	23.93	1.27
Tintri Worker 1	4k 70% Read 100% Random	13036.61	9132.6	3904	50.92	35.67	15.25
Tintri Worker 2	8k 55% Read 80% Random	12288.17	6759.12	5529.05	96	52.81	43.2
Tintri Worker 3	8k 95% Read 75% Random	5944.5	5645.89	298.6	46.44	44.11	2.33

Current Storage Total IOPS: 9740.55
Tintri Total IOPS: 31269.28

For the next test I decided to bring in some more worker VMs to run some concurrent testing. I reused the OLTP/Exchange workload simulations from Tests 1 and 2, and I also added another workload which I believe was the web server option in the I/O Analyzer. Once again this test showed me that both storage systems handled the workloads fairly well but the Tintri simply has a lot more horsepower than my current system. It really shows you the power of flash storage and the impact that it can have on your major applications. My current storage system is a full blown traditional SAN taking up a full 42U server rack and it's probably got about 150ish SAS disks and this 3U Tintri system with 16 disks is blowing right by it on every test.

At this point I had a pretty good idea of what the Tintri was capable of and I certainly showed that it has a lot more power than the storage I'm using today. My next objective was to try a few tests to see just how much throughput I could push through the Tintri system because why the heck not?

Test 4 - Tintri Stress Test w/ 5 workers - 5 Minute Duration

Storage	Workload	Total IOPS	Read IOPS	Write IOPS	MBPS	Read MBPS	Write MBPS
Tintri Worker 1	4k 70% Read 100% Random	8389.25	5872.5	2516.75	32.77	22.94	9.83
Tintri Worker 2	8k 55% Read 80% Random	9139.71	5024.52	4115.19	71.4	39.25	32.15
Tintri Worker 3	8k 95% Read 75% Random	3719.73	3533.8	185.93	29.06	27.61	1.45
Tintri Worker 4	16k 66% Read 100% Random	8548.35	5636.2	2912.15	133.57	88.07	45.5
Tintri Worker 5	8k 20% Read 0% Random	3923.99	784.34	3139.66	30.66	6.13	24.53

`

Total Tintri IOPS: 33721.03

So the IOPS count on this test was a little lower than I expected but still very very good. I re-used a few of the same workload patterns and threw in a few other types from the I/O Analyzer to run 5 concurrent workloads against just the Tintri. Since I only had three hosts to work with I figured that some of the workloads were likely competing with one another so that may account for why I didn't see a significantly larger total number than I did in Test 3.

After posting my first blog entry on twitter Justin Lauer recommended that I try an appliance called Tingle that is provided by Tintri and available for customers on their support site. Much like the VMware I/O Analyzer, this is an OVA template that you deploy into your environment and it can run I/O simulations against your storage. I already had enough good data from the I/O Analyzer but I figured it might be fun to give Tingle a shot too and see how hard it would push the Tintri. I deployed 5 of the appliances and ran their "IOPS" test which executes 20 threads doing random reads and 1 thread doing random writes for a read/write ratio of 71:29. Tingle does not give you the detailed reports that the I/O Analyzer does (you are able to watch the appliance console screen and read the output to get an idea of how each appliance is doing), but my I/O stabilized around 55-60k IOPS and for a very brief period I had the system pushing 75k concurrent IOPS:

I'm honestly not sure how accurate of a test this was and I'm tending to lean toward the numbers that I got from the I/O Analyzer as a bit more accurate as to what I would see in the real world, but mainly what I learned from all of this is that the Tintri's got some serious power boxed up in a relatively small appliance. I think that it's going to be hard for businesses to ignore the performance and ease of use that you can get out of something like a Tintri when comparing against more traditional storage options. The fact that I was able to get this system up and running in very little time and get this kind of performance is extremely appealing to me, and I think it will be to many others as well.

Now that I've got some good data for how my Tintri T540 is going to perform we're moving closer to putting some actual workloads on it! I'm also interested in finding out what kinds of alerting and information that I can get from the Tintri and how their support system works because that's something that can't be ignored when you're looking at new storage. Say what you will about how the support rates from company X or Y, but you know that the big players usually have pretty big support organizations behind their products and the phone gets answered when you really need it to be. I will dig deeper into this and more as my adventure with Tintri continues!

Fun with Tintri Part 1

One of the big things going on in the storage industry right now is the noise being caused by flash storage. Seems like almost overnight companies have popped up and they are putting flash storage almost everywhere. You've got hybrid storage options with flash and spinning disk, all flash storage arrays, host based flash solutions, and probably lots of other types I don't even know about yet. Earlier this week Cisco made some noise when they announced they would be acquiring flash storage provider Whiptail, a company I had never even heard of until I read the press release. This sparked some interesting debate amongst the tech crowd on twitter and various blogs I check out about how this acquisition may fit into Cisco's overall strategy and I suppose that remains to be seen.

So anyway the point is that flash is a pretty big deal right now and it's clear that there are many storage options for the enterprise popping up that are taking heavy advantage of it. Some of these storage options are also moving away from the traditional SAN model and going toward a more purpose built approach or targeting a certain business function. As I run a very heavily virtualized environment with very few traditional physical systems still in operation, I was attracted to Tintri early on in my research and wanted to learn more about them which I had the opportunity to do at this past VMworld. My team did some investigation and talked with some folks out in the industry and decided that it would be worth taking a look at Tintri and earlier this week our VMstore T540 arrived.

Unboxing

The first thing that you'll find when you receive a Tintri appliance is that they do a nice job of packing it up quite well.

Ok so that's not too terribly exciting. Once we got this thing out of the box we saw that the appliance itself is a solid 3U chassis and is very much assembled by our fine friends over at Supermicro, which I have since learned is used by many storage vendors for their offerings as well. I believe I had heard that this was the case at some point but I didn't quite remember until I got it out of the box and noticed the branding on top of the chassis. (Sorry some of these pictures are a little blurry - I'm new at this mess!)

The Innards!

Being engineers naturally the first thing that we did was pop off the top cover and see what's going on inside.

I'm no expert when it comes to case design but it certainly looks like they built this to where people won't screw around with the magic smoke inside these things and I can't say I blame them for that one. It certainly looks well organized inside and they've done a good job to keep you away from the important stuff. We weren't interested in messing with any of the components inside anyway - just checking things out. You'll quickly see on the inside that the Supermicro branding jumps out at you again.

The Bytes

So the T540 system is a 16 drive system with 8 each of solid state and spinning drives. It looks like the HDDs are 3 TB drives from HGST which is a Hitachi company that was purchased by Western Digital (had to look that mess up) and then the SSDs are from Intel. They've rigged up the SSDs into a standard HDD size drive carrier so I wasn't able to see a ton of information about the SSDs themselves without removing the drive and we weren't quite interested in doing all that.

Racking/Cabling

The rails included with the Tintri are fairly easy to get installed which was nice to see. So many times have I purchased various systems and been sent an awful rack mount kit that just makes me want to cry. With the Tintri there's a section of the rail that you have to remove and attach to the system itself and rather than using screws there are some notches on the chassis that the rail slides under and that holds it in place. You may have to use a hammer or whatever you've got laying around to get it in there nice and snug but it goes on without too much trouble. The rails themselves snap into your rack much like the Rapid Rails that I have with my Dell PowerEdge servers. Once you get all of that in place you just line it up and slide the rail sections on the chassis into the catch on the rails on the rack and then tighten it down.

Another nice touch was outside of the standard power cable that you get with most devices (NEMA 5-15 to C13) they also included some adapters to go from the 5-15 connection into a C14 connection which I see commonly used inside datacenters. It didn't help me in my situation (I need C13 to NEMA 6-15P for my PDUs and I keep plenty of extras on hand) but I'm guessing that the guys at Tintri have been in that situation before when you buy a new device and it doesn't come with the right power cables for your datacenter. Thank you for at least giving us some options!

The T540 is a dual controller unit in Active/Standby mode. Each controller has four network connections - two 1 gigabit connections for management traffic and two 10 gigabit connections for data traffic. There's also a slot for another optional NIC that you can add and dedicate to replication traffic but our system did not have that. To have the minimal configuration and still maintain redundancy you'll need four network connections - a 1 GB and a 10 GB connection to each controller. I decided to go ahead and go all out and cable up all 8 connections for a little extra comfort. Naturally I cabled up each controller to two different switches so I would have protection against cable failure and switch failure. There's also two redundant power supplies along with a connection for a KVM dongle so you can access the local console to do the initial config. If you connect up everything it'll look a little something like this:

The story so far..

I haven't had a chance to do much with the system yet but the racking and cabling was fairly painless. In the next few days I'll have an opportunity to start playing with this thing and I'm very interested to see what kind of performance I can get out of it. I hope to share more info as I move through getting this thing up and running and get some VMs running on this guy. Stay tuned!

VMWorld 2013: Part 2/Lessons Learned

So in my last post I summarized a lot of the high points of my VMWorld experience even though I didn't make it through every day. Rather than type up a summary of the rest of the days I thought it might be a bit more beneficial to talk about lessons that I learned from my first VMWorld experience for first timers that may be looking at this next year or in future years. I know that in my preparation I was reviewing tips and lists from a number of notable bloggers and industry folks out there to figure out how to best get ready for the conference and I highly suggest you read those as well to get varying opinions. Without further ado:

1. Wear comfortable clothes - This one was easier for me than I think it may be for other folks who are trying to network to find a new job or opportunity but I wanted to make sure I was comfortable this week. My standard work attire consists of a comfy pair of sneakers, jeans that aren't torn to shreds, and a polo or button up shirt and this is what I wore throughout the entire conference. Comfortable is relative to the person wearing the clothes so this can be whatever you want it to be. I saw people wearing t-shirts and jeans, I saw people in suits, and just about everything in between so odds are you'll be fine with whatever you decide to wear.

2. The food at VMWorld isn't awful - Don't get me wrong, it's not GREAT either but it's certainly edible. I did my best to try to eat Breakfast/Lunch most days at the conference to try to keep my expenses down and most days there were decent options. There are much better options out there so if you have the budget or just don't want to eat the VMWorld food you can do so, but most everything I read prior to the trip was bashing the food so I wanted to ensure I provided my opinion on this one. Plus the meals are a great time to strike up a conversation with other IT pros and chat about the conference or swap IT stories. I had some successes and failures with this myself but I'm glad I made the effort.

3. You can't go to every breakout session even though you might really want to - I booked 5-6 sessions every day that I was interested in going to see and I probably bailed on 1 or 2 daily because there simply wasn't enough time. You'll find very quickly that there is so much to do and see that something is going to come up and you'll likely end up missing a session or two if you book too many. For me, I usually missed a session because I was on the floor of the solutions exchange talking with a customer of a product I was interested in or because I was just super tired and needed to take a break. The good thing is that if you bail on the session that opens up a spot for someone in the standby line who wants to see the session so it balances itself out, but try to schedule the sessions that you really want to see and remember that you can always watch the recordings of the sessions after they are posted online. Which leads me to the next item..

4. Don't take a picture of every slide in the breakout sessions - I seriously could not figure out why people were doing this. The sessions get posted online after the conference and all attendees have access to these recordings, but still I saw phones and tablets going up after every slide change. All I could do was shake my head. Don't be that guy/gal.

5. Take breaks - The days will go by quickly but it's best to remember to stop and take a break every now and then so you don't wear yourself out. This could be stopping by the hang space for a bit, or just arriving at a session a bit earlier than you planned and taking a rest. Going along with this I carried Water/Snacks with me at all times in my VMWorld backpack and that was extremely helpful. It can get a bit hot in the solutions exchange especially with all of those people in there so it helps to have the water nearby. There's also plenty of water dispensers set up if you don't want to carry it around all the time.

6. Use social media - One of the things I enjoyed most during the conference was following along on Twitter with what the big names had to say, but also what people thought of the sessions that I was in. Every session has a session ID which makes for an easy hash tag and I did my best to use these to comment on every session that I was in. Some sessions had a lot of activity on Twitter this way, and others did not which was somewhat disappointing after experiencing some of the more socially active ones. I was also checking badges for twitter handles and I was sad that I didn't see very many amongst the general attendees. Perhaps they weren't on twitter or they just didn't want to share and that's fine either way, but I was hoping to find some more folks to follow outside of the big names and I didn't get a whole lot of them.

Side note from this one: Eric Shanks did a great statistical look at Twitter usage from this year's VMWorld here.

7. Engage the presenters or other big names - I'll admit I didn't do as good a job with this one as I would have liked, but I did better than I expected. Most of the folks are very approachable and will talk to you or just say Hi if that's all you're looking for. One of the people I had a chance to meet and speak with was Simon Long and he was very nice to talk to. His blog was a great resource when I was starting out in virtualization and prepping to take my first VCP exam (Version 4) and I wanted to thank him for putting all of that together. I also ran into Scott Lowe on my way out on the last day and I just said Hi to him and he was very nice and offered me an extra Spousetivities t-shirt for my wife which was awesome. This is one I hope to improve on next year.

8. Have extra room in your luggage - Airline bag fees suck and the last thing you want to do is have to check an extra bag on the way home. You are going to come home with more stuff than you showed up with unless you just completely avoid the solutions exchange which I do not recommend. I ended up coming home with like 17 or 18 new shirts, some bags, plus all the little trinkets like buttons and stickers and such that vendors like to give out. I brought a larger suitcase than I normally travel with for this specific reason and I am VERY glad that I did. I was able to get the extra backpack and all this stuff into my bag which just made it easier to deal with when I was going home (although the bag was quite a bit heavier I do admit). Most airlines allow two carry-ons and if they run out of room in the overheads they'll check it for you for free so that's another good way to handle it if you prefer.

9. Let your vendors know you're going to VMWorld - Most of them are throwing parties or putting on events during the nights of the conference and they will usually extend you an invite if they know you're going. I was fortunate enough to attend the Netapp MVP event which also took place at AT&T Park the night before the VMWorld party but I didn't get an invite until a few days before the conference because my rep didn't know I was going till the last minute. Netapp put on a pretty awesome party and I got to meet some sports icons including Brandi Chastain, Joe Morgan, and Bret Saberhagen:

So let your reps know you'll be there and you never know what you might get to check out! Vendors on the solutions exchange sometimes can get you invites to their events if you stop by and ask them about it as well but I was already booked up before I even arrived.

10. Enjoy it! - It is a professional event and I was primarily there to learn and grow as an IT pro, but it's a heck of a lot of fun too. Take some time to "smell the roses" per say and take it all in and enjoy it. It really is a blast!

Hope this helps some folks that will be heading out to their first VMWorld in future years. If you remember nothing else from what I wrote here do remember #10 - enjoy the experience! I know I did and I'm already counting down to VMWorld 2014. Just 360ish or so days to go!