WestPoint · Hardware infrastructure
The thirty days of recording were never thirty.
Servers, storage, backup power, cabinets and plant room: what holds a security system up from underneath. It is sized with a calculation and checked with the system running, not on the datasheet.
The day you have to pull the recording.
And it turns out there is nothing left of it.
Somebody asks about a lorry that came in three weeks ago. The contract said thirty days of recording, so you go and look and it is not there: the system keeps eighteen. Nobody has changed anything, nobody has deleted anything and nobody had noticed.
What happened is easy to explain and nobody explained it in time. The thirty days came from a calculation done in winter, with twelve cameras and with the average bit rate of each one. Then six were added, the quality of two was raised to read number plates, and summer arrived with fourteen hours of daylight. The drive is the same and the sums no longer work.
A number of terabytes is not a number of days. The days depend on how many cameras there are, how much detail each one has, how many hours something is moving in front of them and whether it records all the time or only when something happens. Change one of the four and the days change.
Sizing is not picking the model at the top.
A video server does three different jobs, and each one asks something different of the machine.
The first is receiving the images and writing them to disc: not difficult, but it never stops, and what it asks for is endurance. The second is displaying them, and that is the costly one, because to see sixteen cameras on a screen you have to decompress sixteen videos at once. The third is understanding what it sees, and that one is not fixed by more of the same power: it is fixed by the part that does that job, which is a graphics card.
That is why a server well chosen for recording chokes on opening day, when six people open sixteen-camera mosaics at the same time. It was not badly bought: it was sized for the job nobody sees.
And there is an earlier decision that changes the rest: whether that video will ever be analysed. Leaving room for that from the start costs very little; doing it afterwards means changing the server. Infinity Neural is in the same house, so we always ask the question, even though the answer is almost always no.
An array of drives is not a backup.
It is the most repeated confusion in this trade, and the one that leaves most people without their recording.
A drive array with redundancy protects against one specific thing: a drive breaking. It breaks, the system carries on working and the part is replaced without shutting anything down. That is a lot, and it is all it does.
It does not protect against somebody deleting the recording, nor against it being deleted on purpose by somebody who had permission. It does not protect against the fire that takes the cabinet, nor against the theft of the recorder, which is the first thing anyone who knows what they are doing takes. And if what is being written is wrong, it is written wrong to every drive at once.
There is also a price to rebuilding that nobody talks about. While the system refills the new drive —hours, or days on large arrays— the other drives work flat out, and that is exactly the moment when the second is most likely to go. If it goes then, there is nothing left to rebuild.
So storage is decided with two questions: how long does this keep running if a part breaks, and what happens if the whole site disappears. The second has no technical answer — the client decides it.
The first five minutes without power.
Almost every installation has backup power. Almost none knows how many minutes it gives with what is plugged into it today, which is never what was plugged in on the day it was calculated.
There is one mistake that comes up constantly: backing up the recorder and not the switches. The recorder stays on, perfectly, recording nothing — because the cameras are powered from the switch and the switch is off. If something has to keep working through a cut, the whole chain has to keep working: cameras, switch, recorder and the workstation it is watched from.
Then there is the order. Five minutes of backup are enough for a flicker to go unnoticed; two hours is another decision and another price. And if the cut is a long one, someone has to have decided what shuts down first and what shuts down last, and that the server shuts itself down cleanly instead of losing power halfway through a write — the most common way of ruining a recording.
And batteries age without saying so: a six-year-old unit gives half what it used to and keeps showing green lights right up to the day of the cut. That is why the runtime is measured with everything running, and measured again every year. It is half an hour that almost nobody spends.
The cabinet is in the cleaner's cupboard.
We have found it there many times, and almost always with the door shut.
A recorder and a switch together give off heat all day. Inside a closed cabinet in a room with no ventilation, in July that cabinet goes past forty degrees. Drives are the first thing to die of heat, and not suddenly: they last two years instead of six, and nobody connects it to the temperature.
So the cabinet is a decision, not somewhere it happens to fit. How much free space it leaves, where the air comes in and goes out, what filters it if the environment is dusty —a joinery unit is not an office— and whether it needs cooling or just moving air. Plus a thermometer that raises an alert, which is the only thing that turns a future problem into a warning.
Inside there are two more things that look minor. The cabling: forty identical patch leads, crossed and unlabelled, turn any change into an hour of risk, because to move one you have to touch thirty. And the key, because the whole building's security system lives inside that cabinet, and getting to it is usually a good deal easier than getting into the warehouse the cameras are watching.
Here, off site, or both.
All three work. What does not work is choosing without knowing what you lose with each.
Everything in house is what almost everyone has: the video never leaves the site, it works even if the line goes down and nobody outside touches the images. In exchange the equipment is yours — you pay for it, maintain it, cool it and replace it —, and if the place burns down or somebody takes the recorder, what was inside goes with it.
Everything off site removes that work and adds a dependency: an upload connection that carries all the video every hour of the day —far more than people imagine— and a fee that never ends. For four cameras in a shop it can be the best possible decision; for eighty in an industrial unit, the sums do not work.
What works on large installations is almost always splitting it: record everything in house and send off site only what has to survive the place — what set off an alarm, what has to be kept for an open case, or a low-quality copy of the cameras that cannot be lost. And there is an argument that is not technical and decides a fair amount: where the images are and who can look at them. In banking that is not a preference.
Hardware ages to a date.
And the date can be known two years ahead, which is the difference between a quotation and an emergency.
- The drives in a recorder are chosen to write without stopping, because that is what they do: an office computer drive works and lasts far less time, because it is designed to sit idle most of the day. And one that has spent five years writing is at the end of its life even if it works: replacing them on a schedule costs money, replacing them when they fail costs the recording from those days.
- Updates are applied with a plan and with a way back: a good many of them close holes people get in through, and the other half of the problems on a stable system arrive the day somebody updated at midday.
- Every piece of equipment has a date when it stops having support and spares. Knowing it two years ahead turns a replacement into a line in next year's budget; not knowing it turns a fault into an emergency with whatever is to hand. And a spare that takes three weeks to arrive is not a spare: which parts are worth keeping to hand is decided at the design stage.
- A seven-year-old server is not replaced with another one the same, because it no longer exists. That is why the documentation is worth so much: with it the new machine is in place in a day; without it, by trial and error.
What we do not do.
We do not give days of recording without saying what calculation they came from: thirty days with these cameras, at this quality and with this much movement in front of them. If six get added tomorrow it is no longer thirty, and that gets said on the day they are added.
We do not squeeze a server to the limit to make it fit the budget: a machine running at ninety per cent on handover day has no room for what is going to be added, and something always gets added. And we do not put a cabinet somewhere we know it will not survive August: if there is no suitable place, the honest thing is to say one has to be built.
And we do not call a drive array with redundancy a backup. We are writing it here because it has been said to us, in a meeting, in front of the client — and the client believed it.
How you check it yourself.
Five checks, this week, without touching anything.
- Ask for a recording from exactly as many days ago as you were promised. Not from a week ago: from the whole period. It is the check that produces the most surprises.
- Ask which part takes longest to get hold of if it breaks today. If nobody knows, you will find the answer out on the day it breaks.
- Open the cabinet on a hot day and put your hand in. If the air inside is warm and still, you already know what is going to break first.
- Ask how many minutes the system holds without power and whether anybody has measured it with everything switched on. A number that comes off the box is not a measurement.
The other seven areas.
Everything else ends up writing to these drives and drawing on this power. That is why the hardware is sized with the project in front of you and not at the end.
The sixteen brands that hold this up underneath.
Servers, storage, GPU, power and cabinet. Each one with where it fits, where we would not put it and what really decides the sizing — which is where the money is lost, not in the brand.
What people ask about the equipment.
The ones that come up most. There are 25 on this subject, and the other 19 are in the question index.
All the questions, by subject▸How many days of recording fit on my system?
Whatever comes out of the calculation, and there are four variables in it: how many cameras there are, what bit rate each one produces, how many hours a day it actually records and how much space is left usable after drive redundancy and the system itself. Change one of the four and the days change.
That is why a number of terabytes is not a number of days, and why the figure that appears in many contracts —«thirty days»— is a consequence, not a feature of the equipment.
And the honest check is not looking at the configuration, which will say whatever was put into it. It is asking for a recording from the oldest day of the promised period and watching it. It is the check that produces the most surprises.
▸How is the days-of-recording calculation really done?
You turn the bit rate into a volume and multiply by the days. There is one equivalence you need: one megabit per second sustained is about ten point eight gigabytes a day. With that you can do the sums with a pencil and paper.
An example, and it is only an example: twenty-four cameras at an average of four megabits per second is ninety-six megabits per second, which at ten point eight gigabytes a day per megabit gives a little over a terabyte a day. Thirty days would be some thirty-one terabytes of video. On top of that you have to add what drive redundancy takes and leave free space so the system has room to work, so the actual purchase is a good deal larger than the figure from the sums.
The weak point in all of this is those four megabits. If they come from a datasheet, the calculation is born wrong. They are measured with the cameras installed, on the real scene and at the worst hour, which is at night with movement in front of them.
The storage calculation, step by step How a project is sized
▸Why do I have fewer days of recording than I was promised?
Almost always because the installation changed and the calculation was not redone. Cameras were added, the quality of two was raised so number plates could be read, summer arrived with fourteen hours of daylight and far more movement in front of them. The drive is the same and the sums no longer work.
The second cause is the original calculation being done with the cameras' average bit rate instead of the real one. A system calculated on the average keeps the promised days on the quiet days, which are exactly the ones nobody needs.
It is not an equipment failure and nothing has been deleted: the system keeps dropping the oldest, which is what it is meant to do. What failed was not redoing the calculation. That is why every time a camera is added, someone has to say how many days are left from that moment on.
Why the thirty days were never thirty What each system inspection checks
▸Does recording only on movement save as much as it looks?
It saves a lot where nothing happens and almost nothing where something does, which is usually where it matters. A warehouse closed at night can drop enormously; a street with traffic or a loading bay during working hours record practically continuously, so the expected saving never appears.
And it has a cost you have to accept: the detection has to trigger before the interesting thing happens. It is set up with a few seconds of pre-recording so the start of the scene is not lost, and even so a badly tuned detection leaves things out. In an area that genuinely matters, that risk is not worth the disc space it saves.
What we usually do is mix the two: continuous at a low frame rate, with the quality raised when there is an event. That way there are no holes in the timeline and the drive lasts a good deal longer.
What stored video really takes up What IRIS Neural detects in the scene
▸Why are drives from an ordinary computer not good enough?
Because the job is different. A drive designed for surveillance is built to write without stopping, every day, with several streams coming in at once and with the vibration of other drives spinning alongside. A desktop one is designed to sit idle most of the day and work in bursts.
Put in a recorder it works, and it lasts far less time. They also behave differently when there is a read error: a desktop one can sit retrying for a long while, and the array controller reads that as the drive having died and throws it out. That alone is enough to set off a rebuild nobody asked for.
And there is a part that does not depend on the model: a drive that has spent years writing without stopping is at the end of its life even if it works. Replacing them on a schedule costs money; replacing them when they fail costs the recording from those days.
The drives and arrays we install Seagate drives for continuous recording Western Digital drives, brand profile
▸What type of RAID suits video?
It depends on how many drives there are and how much you can afford to lose. With few drives, a scheme that survives one failing is usually enough. With many, it is worth one that survives two, because the real risk is not one drive failing: it is the second failing while the first is being rebuilt.
Redundancy is paid for in capacity, and that has to be counted at purchase, not discovered at formatting. A drive array always offers less space than its drive labels add up to.
And there is an earlier decision that gets little thought: splitting into several medium arrays instead of building one huge one. Rebuilding a very large array takes far longer, punishes all its drives at once and leaves the installation without a safety net for longer.
How drive redundancy is built NetApp arrays, brand profile Synology storage, brand profile
Get started
How many days of recording do you really have?
Tell us how many cameras there are, what gets recorded, how long it has to be kept and where the equipment is. With that we do the sums and tell you whether what is installed delivers what you were promised. Sometimes it does, and we say that too.