Recently, I was having issues with internal DNS names, I had spent some time using the .local top level domain name (TLD).
I’d been content with the .local domain name for a good while, because it worked on Windows, but I found that when using the domain names on Linux and Android it would not work in a browser.
It turns out .local is used for in link-local networking. Often for something called mDNS, which I admit I don’t know a lot about.
So if you have been content connecting to your internal domains using ca.local, octoprint.local, proxmox.local I have bad news for you, these domains aren’t going to work much longer.
Windows Command Prompt pinging proxmox.lan and proxmox.local
Changing .local Domains on EdgeRouter
What I did to fix this was login to my EdgeRouter X and change all of the references from .local to .lan. Although I cant promise .lan won’t one day be victim to the same fate.
If you have an EdgeRouter X its easier to bulk edit the domain names using the config tree, although make a note of your configuration, because it won’t be preserved if you alter the parent node of each configuration item (so you’ll loose aliases).
Config Tree > system > static-host-mapping > host-name
> Host name for static address mapping
Remember to commit your changes.
Can I alter DNS for my devices without a DNS Server?
If your DNS isn’t under your control, you won’t be able to configure how it responds to queries, there are some hacks and tricks to get around this, such as editing your hosts file.
Using the hosts file works fine, but DNS can become complicated quickly. Especially if there are many devices.
For best results it is best to configure your DNS as required, rather than making edits to each device’s configuration.
Editing your hosts file may also not be possible on Android devices for example, or TVs.
You don’t have to use .lan for a TLD, it is probably best practice to register a domain, and then use everything you need as a subdomain of that domain, as is typical of larger networks and allows for segmentation.
I decided it was a good time to learn docker and actually make a project that uses it, so I created ICO Security Trends, a small and unique dashboard which uses the UK ICO published spreadsheets to produce graphs and insight into the data.
I thought I would include some of my findings which are not immediately evident on the BI Dashboard they provide,
UK ICO Incident Security Trends
Categorisation on incidents described as ‘Other non-cyber incident’ has declined from 2019 to 2022. Roughly on average there are 750 incidents a quarter for ‘Other non-cyber incident[s]’, while ‘Other cyber-incidents’ remain fairly constant at around 60 a quarter.
The ‘Other non-cyber incident’ is generally too broad and should potentially be broken down. Insights into trends in this area are potentially being missed.
Ransomware disclosure has increased since 2019, which concides with general industry concensus.
There’s a lot more to it, but I thought I’d get it out there already,
Capital Expenditure and Operational Expenditure are spending methodologies that focus principally if it is better to use a cloud service, on premise or a hybrid method to deliver service to users.
Please note this article is written mostly within the domain of IT infrastructure, however there are aspects of this subject which are also applicable to accounting methodologies for Capital Expenditure and Operational Expenditure.
A limited analogy of Capital Expenditure and Operational Expenditure is to think of Capital Expenditure as renting infrastructure and Operational Expenditure as a purchasing your infrastructure. There are advantages to both types of financing your infrastructure.
Capital Expenditure (CapEx)
Capital Expenditure is the up-front spending on physical infrastructure such as hardware,
Examples of Capital Expenditure
Networking Equipment
Servers, Spinning Disk, SSD and Compute
Racking Equipment
Climate Control Equipment
Cabling and Termination (Patch Panels, Ethernet Pattress Boxes)
Software
Monitoring and Disaster Recovery
Land or Property
Generators
Security
Capital Expenditure usually has a large inital cost as a new project is delivered, over time the cost of the infrastructure will remain mostly constant until hardware failures, obselesence and improvements to methodologies cause changes to configuration.
Renewals
Expansions
Upgrades
New Infrastructure
Advantages of Capital Expenditure
Greater control over fixed assets, which can be liquidised if required, this applies not just to IT equipment but warehousing, property and vehicles.
Property and equipment can be kept physically close to users to provide faster, cheaper throughput. Access to resources that could be internal do not rely on the WAN.
Low cost to bandwidth for even medium sized organisations when transferring locally. Low or no cost transfer between hosts.
Upgrades and downgrades to hardware is generally easier than cloud based solutions. Cloud based solutions generally do not provide an easy method of scaling down infrastructure.
Issues with the host can be identified by onsite skills rather than second or third party infrastructure teams.
On premise infrastructure still workswhen wider services like Cloud solutions, ISPs or links between sites go down.
Capital Expenditure is generally better for organisations with a lot of techical debt, because applications can keep running for years with little cost. When reaching capacity older appliances can be decomissioned to free up space for newer hardware or software.
Generally heavy compute or IOPS (Transfer between appliances) is best suited on premise, because the cost will be lower than cloud.
Disadvantages of Capital Expenditure
Large initial cost to start a project.
Large cost when appliances need refreshing/upgrading.
Growing or Expanding businesses will require more assets, including space and electrical infrastructure to support the requirements of the business.
Redundancy and SLAs are on the business, power outages or natural disasters can cause business closing consiquences.
Large monolithic applications can cause technical debt.
Generally the value of the assets you purchase decreases over time and with wear.
Operational Expenditure (OpEx)
Operational Expenditure is different in that the cost of the asset is not included at the start but you continually pay for it throughout the lifetime of the project.
Operational Exepnditure is also more dynamic as you can pay your provider to scale up and down the instances horizontally or vertically as needed, increase compute or storage, or increase the number of nodes required.
Examples of Operational Expenditure
Storage Cost
Administrative Expenses
Maintenance
Subscriptions and Licencing
Generally it can be easy to decide which option is cheaper over the lifetime of a project if an on-premise business has already invested in the surrounding infrastructure, however increasingly cloud providers are offering lower cost incentives such as lower cost products than a traditional virtual machine or cheaper hardware through their platform.
Advantages of Operational Expenditure
Products are generally available quicker for purchase for project managers than on-premise, providing a more agile IT strategy.
Expenses are generally at every billing cycle consistently, rather than an initial cost for hardware and then a smaller maintenance cost.
Generally you pay for consumption where you only pay for the resources used, however some providers will agree to a discount if a commitment is made for resources over a longer term.
Costs can be predicted and generally billing can be itemised easily, which helps in understanding risk.
Some IaaS and most SaaS solutions will patch and maintain infrastructure, such as databases without much input from the product owners.
Disadvantages of Operational Expenditure
Generally costs for highly transactional applications can be higher than on-premise solutions, such as compute or IOPS intensive workloads.
Decisions may be set with a short term view and poor cost analysis.
Availability of skilled professionals is not easy.
Corporate Networks are highly thought out and well-designed critical business infrastructure that can span many buildings or geographies. The more complex an organisation is, the more expansive and multi-format the network can be.
A Corporate Network will often have an acceptable use policy and may monitor its usage.
D-Link DGS-108 Consumer Dumb Network Switch Corporate Network Server Closet
Features of a Corporate Network
Many corporate networks utilise additional benefits that home or small business routers usually are not capable of, such as;
Quality of Service or QoS is a layer 3 network technology that can prioritise (or more importantly de-prioritise) traffic by application, such as streaming, game services or file sharing.
Traffic Shaping is a bandwidth management tool to slow long running or high bandwidth downloads to prioritise other activities and ultimately restrict high utilisation on the network by a single client. This is most useful where bandwidth externally is limited.
VPNs (such as L2TP/IPSec or Wireguard) or SSL Tunnels (SSTP) allow corporate networks to link together across global infrastructure, SSL Tunnels can ensure that all data accessed by clients is encrypted by the link itself, so that any HTTP traffic for example must ultimately first travel SSL encrypted to the VPN routing appliance or provider.
VLANs can segregate and isolate riskier traffic as well as limit chatter or prevent sniffing ports. VLANs can also by separated by different subnets or network classes to protect, prioritise or isolate IT infrastructure and management from users. For example many switches have a management VLAN to prevent end-user clients re-configuring or accessing the management portal for the switch itself.
IPv6 is a relatively common new link format however some organisations are starting to implement IPv6 in their infrastructure in preparation for the switchover. Personally I believe this will not be a requirement for some time.
Content filtering and Proxying is used in organisations to protect valuable data and users from compromise and exfiltration. Some organisations require a proxy to reach external services and most implement some form of content filtering, generally for productivity or traffic management purposes.
DNS or Domain Name System servers can provide internal network resources resolvable and recognisable addressing for internal services. Most enterprises use DNS with Active directory through windows server domain controllers so that their Windows clients can take advantage of resolvable network names for windows machines.
Features of a Large Corporate Network
Larger Corporate Networks, ones that can encompass tens of thousands of devices or more could be considered large and may take additional setup, such as;
Load Balancing can be used to balance demand to external or internal services like internal enterprise applications or highly available applications that are business critical.
iBGP Routing or Border Gateway Protocol is usually only required for extremely large networks. Where routing and network policies are likely to change. BGP Routing is generally only required for carrier ISPs or enterprises dealing with internet infrastructure. For customers, due to the premium on network devices, the requirements of the networks used by enterprises and organisations are generally less than BGP can facilitate and BGP is not supported on smaller SOHO (Small Office/Home Office) networks.
Corporate Network Internal Services
DNS or Domain Name Systems
You may wonder how companies and other organisations are able to utilise top-level domain names that are not typically available on the internet, such as example.local and subdomains for a real domain, such as internal.example.com where internal.example.com is not a real external subdomain.
This is possible through many technologies and can incorporate many aspects to enable additional features like trusted SSL and network-level authentication or windows authentication to provide a relatively normal experience for end-users while being completely inaccessible from external networks.
SSL or Enterprise Trust
Even consumer routers often provide the facility to reserve DHCP addresses and register DNS names and aliases, but providing trusted SSL is accomplished through using either,
A local, trusted SSL certificate signing authority, with the organisations root or supplementary SSL certificate trusted by clients.
A real, actual trusted wildcard SSL certificate for a subdomain of the organisation. This is less common as it would require the same certificate to be on every application.
Network Segmentation and Isolation
A Corporate Network may utilise Network Segmentation to isolate external clients from internal applications or require a VPN to access. In this case, rules on the router allow inter-VLAN communication and routing table rules to allow communication with clients. Some networks may implement a zero-trust architecture in their network access.
Network segmentation restricts access to different services based on rules to help protect an enterprise from enumeration and the exfiltration of data, as access to the network is only possible through opaque rules that will make data transfer over the mediums allowed difficult. For example, access to a public server on a trusted LAN through a direct connection over SSH port 23 may not allow access to web-based interfaces internally such as port 80 or 443 as network rules prevent access, usually by dropping packets.
Many organisations may utilise these technologies in conjunction with an SSL proxy to provide legacy applications with an HTTPS frontend to a web server that is not configured for SSL, as access to the application web server would be restricted to only allow traffic through the proxy.
VPNs and DirectAccess
DirectAccess (similar to an always-on VPN) for Windows or VPN services like L2TP/IPSec enable corporate networks to be spanned over different environments, such as;
Field Engineers who rely on access to internal databases for parts or documents.
Mobile Devices and Tablets for reading email remotely.
Work from Home Deployments (WFH) for office employees who need access to shared drives and groupware.
Satellite or Remote Offices can deploy over the VPN to ensure a consistent experience for employees who travel.
Otherwise insecure environments, like coffee shops can be used as internal services will be accessed over the VPN and not directly over the internet.
Customer Premises where interfaces required on site can be relayed to internal networks at the origin organisation.
VPNs once configured with credentials can be utilised to provide network access as though they were direct clients of the VPN router, which could be placed in a trusted part of the enterprise and provide the typical trust, filtering and proxying required by the organisation configuration.
VPNs can often disconnect at work because there are packets not making it to the VPN provider. The simplest method to rectify this is usually by using an Ethernet cable.
Corporate Network IP Schemes
Unlike a public IP address with a single home network-attached, a corporate network may take advantage of using many IP addresses, networks and physical links to their ISP to provide a more robust and uniform experience to users.
Almost all corporate networks will use VLANs and network subnets to distribute their client environments to isolate services for example, a computer lab in a school vs a teacher network, or an open WiFi network at a restaurant compared to a private one for POS (Point of Sale) terminals.
Generally, most enterprises use the 10.0.0.0/8 IP CDIR block, using different subnets for different kinds of devices. Using the traditional 256 contiguous class C network addresses 192.168.0.0/16 range may not provide enough IP addresses for some larger deployments. (65,536 possible clients).
Corporate Network WiFi
Generally, Corporate Networks used to be a closed ecosystem, where only trusted devices and non-enterprise owned equipment was not present, this is no longer the case.
Rather than use combination Routing and Access Point devices like a home router, enterprises utilise extensive commercial WiFi Access Points that can provide access to numerous clients and can be distributed through the locations the organisation resides, like buildings and restaurants. Using dedicated hardware like Access Points enables the use of specialist configurations, like access point hopping for clients and PoE for easier installation and unification.
Some newer WiFi networks can also provide certificates that can be used in the organisation to access internal resources over SSL.
When I got my Sennheiser 350BT when using games like Warzone or on phone calls on Windows 11 I found I could hear myself and my surroundings played back to me.
I spent a good while trying to disable it so I could no longer hear myself so I am posting it here. There are two possible problems so make sure to check both,
You need to disable sidetone in the Sennheiser Smart Control App. This will probably require you to update your headphones to the latest version.
You need to disable the “LE-HD 350BT Hands-Free” driver in Windows.
Disabling Sidetone in Sennheiser Smart Control
To stop hearing yourself on phone calls, open the Sennheiser Smart Control app then find your headphones and make sure they are connected, and then you should see the setting by clicking the cog in the top right, make sure to set it to 0%.
If you don’t see the sidetone setting the the app, update your headphones as below and follow the instructions for windows below as well.
Once it is set to zero, you shouldn’t hear yourself on phone calls at all, this however is not the only fix required for Windows, so follow the Windows 11 steps as well if you can still hear yourself.
Updating the Sennheiser 350BT
If you don’t have the option to disable sidetone, you may need an update for your headphones if they are not running the latest version.
Pair your headphones to the phone you are using, and install the Sennheiser Smart Control App. You should see a message like this in the app.
If your headphones already have the latest version, you can skip this step.
Sennheiser Smart Control update screen
I found this process to be really really difficult, because it took about 5 or 6 tries to get right, and when you’ve never done it before it can be extremely annoying. I even tried on a different phone before it working when I went back to the device I first tried. As a general rule I found,
Do not leave the app during the update.
You must not play any audio, at all while the headphones are updating.
You must not allow it to pair with any devices other than the phone you are using to update them with.
Otherwise it will not work and you will see a message like this.
Sennheiser Smart Control Something went wrong
If you see a message like above on Android, simply keep trying to get it working. It took me about 6 tries to get it to work.
And eventually I finally saw this screen,
After you have updated your headphones, you should see the setting as above.
Disable the Hands Free Audio Driver in Windows 11
If you still hear yourself in the headphones, I found this was only present in Warzone and Discord. You need to disable the Hands Free Driver for Windows. Note I am running Windows 11 and found there were two driver locations.
Audio inputs and outputs > “Headset (HD 350BT Hands-Free)”
Sound, video and game controllers > “LE-HD 350BT Hands-Free”
In Device Manager (Search in Windows for “Device Manager”), right click both (as in one in both categories, “Audio inputs and outputs” and “Sound, video and game controllers”) drivers that are labeled hands-free for your headphones and select the option to disable them. A reboot will be required most likely to stop hearing yourself.
Audio inputs and outputs > “Headset (HD 350BT Hands-Free)”Sound, video and game controllers > “LE-HD 350BT Hands-Free”
Verify You are no longer Hearing Yourself
After a restart if you load up Warzone or call over Microsoft Teams or Discord, you should both no longer be hearing yourself and when making calls on your phone, you should no longer hear sidetone.
If you still do, make sure the app setting is set to 0% and the drivers have been disabled. I would not reccomend uninstalling the driver outright for Windows 11.
When working on a new web application there are some crucial aspects to your application security that all developers should follow.
This applies to both a production and test environment. Just because an application is in-test or not production-ready does not excuse poor security. There are a few examples of where even ‘secure’ environments have been exploited through their test systems.
Secure Development Environments
Should not use real-world data and should rely on faker or placeholder data. This can be more time consuming for agile teams as the data may change over time, which is why your ORM models should migrate to support the latest faker schema.
Should be isolated entirely from the production environment and should not be placed in the same realm of systems. Your environment configuration should be independent from your secrets configuration and of course neither should be versioned. Whenever there is a need for an environment or secrets file to copy from, it should be made available in the documentation as an example and should not use real credentials.
Application code should be clear and understandable with well documented features, when you use a library for authentication it should be well-maintained and you should always use secure and up-to-date libraries for your application.
Your development and deployment pipeline should not have any secrets passed in via terminal command. Logging and event monitoring may inadvertently record these credentials, which is insecure as logging may not always be a privileged activity.
Your source code should not be considered a weakness if exposed, in your organisation (or outside it) you should practice an open source initiative. If your code base were to be exposed, it should not be too detrimental to your security. This principal doesn’t totally apply to blue team defence or anti-cheat because detection methods are hard to prevent exploitation, however this can be mitigated by having a narrow domain of allowed activity.
At all avenues, SSL and TLS should be used as well as encryption, both in transport and at rest.
How Do I Know If My Web Application Is Secure?
Determining if your web application is secure can be hard to do if you are not a developer for the application in question however, there are some basic techniques you can use.
Do not overly expose your application, if it is only used internally then make sure it only is available through the company intranet or local area.
Do not expose parts of the application that are internal only to the application itself, if the application uses a MYSQL database, there is no purpose in exposing the MYSQL database to the internet if the client only interacts with the webserver.
Do not expose APIs and application endpoints that are internal.
Do not allow anonymous authentication to applications or use shared credentials.
Log and monitor behaviours, especially database queries and crash behaviour.
Make sure your application is up to date and supports the latest version of your web-server and databases.
When using proxy technologies, make sure to follow the proper domain rules applied by the web server and make sure sessions are properly catered for when using load balancing.
Use trusted SSL technologies and transport.
Do not use credentials for the application that allow for lateral movement throughout your environment, isolate public services not only through networks but also by authentication.
Moving your applications to the cloud can be a polarising task. Some believe that the cloud is the future and the value add outweighs the work involved. Others believe that on premise solutions are best for users and data.
Vendor Lock-in
Although vendors like Azure may offer greater features or added value than traditional on-premise services, there can often be features that PaaS services offer that are not available in other solutions that can increase the total cost of ownership.
Features are not the only drawbacks to cloud solutions, many cloud platforms can offer additional infrastructure or architectures that can make lift and shift harder or impossible. This is especially prevalent for IT teams that do not have software development in their departments.
Downtime
Many cloud services, especially global solutions like AWS, Google Cloud or Azure have had downtime in the past and although they provide service level agreements to their services, they have had downtime that can span many hours.
In addition, configuration changes and DNS are a big factor to consider. Managing trusted IP addresses and DMZ’s over the internet can be difficult to implement effectively. Many cloud solutions are better than what in-house services can offer but resiliency and accessibility becomes an overhead that requires consideration.
Many cloud solutions offer backup technologies and solutions with low import fees but high exit fees when transiting data.
Skill Gaps and Lack of Expertise
As new cloud solutions develop, it is becoming increasingly difficult to learn new skills for all platforms and effectively transfer them from one provider to another, product names and technologies are complex to learn and certifications for one cloud platform may not always apply to others.
Cloud technologies on the whole are here to stay and rapid advancement in technologies can be costly to small business to implement. Many companies are using automated solutions like backups and Azure AD to manage systems that previously were done in house, this can make break-glass protocols or access to systems where the administrator is no longer present difficult.
Bandwidth and Data Transfer Fees
Saving money is a common driver for business decisions and cloud solutions offer good value for services once in-house, however the cost of moving data around the enterprise can be expensive when a hybrid approach is adopted. This is especially prevalent for large transfers of data off the cloud or public internet.
Many cloud solutions can also deliver data faster via CDNs or through local appliances to alleviate this cost, however this requires some investment and can lead to vendor lock-in it can lead to better globalisation and work from home approaches to working.
Choosing the right service can be complex, such as choosing a hosted VM over a PaaS solution like a database server over a managed database can be hard to estimate best value.
As products and services grow over time, these costs will become more difficult to manage and estimate as requirements by users increase.
Incompatibility
Many enterprises choose a hybrid approach when moving to the cloud, offering services once only internal to the enterprise – now available through the public internet makes security a greater consideration. Older applications deemed acceptable risk or air-gaped may no longer be supported by surrounding infrastructure or end user clients.
Many enterprise applications may not be built for the cloud or may not support security technologies the cloud requires when working with externally facing services, in particular patching and maintenance to applications no longer supported by vendors.
Applications that are depended on by thousands of users may see peaks or dips in demand during the day and managing the cost of running the infrastructure can be challenging. Scalable applications are applications that are able to increase their resources to serve requests as they come.
What types of Scaling are there?
There are two basic examples of scaling for applications,
Vertical Scaling, where an application’s resources are increased due to demand, such as increasing the RAM available to an application host. Vertical Scaling is also sometimes referred to as scale-up.
Horizontal Scaling, where an application is spread across different nodes in a cluster. This is most appropriate for applications that require a lot of resources. Horizontal Scaling is also referred to as scale-out.
Scalable applications have many uses, including;
Allowing cost reduction during low utilisation by scaling down clusters or pods serving the application.
Improving quality of service during peak load times by scaling up horizontally or vertically by utilising resources made available to the application autoscaler.This will ensure that your application always has the right amount of capacity to handle the current traffic demand.
Faster processing as features can use the optimal storage technology for data and information.
Best Practice for Scalable Applications
Applications usually are best compartmentalised into components for both design and maintainability, monolithic architectures for code bases and applications have caused application elements to become obscure and entrenched, using a more distributed approach from the start can reduce cost and technical debt as components can be re-written or swapped out.
Splitting transient application functions in to their own libraries or APIs can allow the front end and back end to operate independently, this can allow processes that take time to be acted on based on events rather than cause waiting or processing as a batch.
Data storage should be independent to the service that owns the data and therefore should use the best storage type and mechanisms available such as caching or streams of data. Slow returns should be accounted for and independent from the user interface.
Applications should not be strained by resources, whenever possible applications should perform the same for every user or function regardless of workload or work-type. Rather than wait for functions to complete, have a common goal of eventual data integrity.
When implementing concepts like micro-services you should endeavour to ensure standard practices like a common set of languages or behaviours to improve maintainability.
Complexity can sometimes be harder to manage than an equivalent monolithic application, even though each function should be simpler.
I recovered my data from AWS S3 and all I got was this lousy bill.
Aidan – Alternate Headline.
TLDR;
One of my hard drives failed, I thought I’d try to recover the valuable 400GB using ddrescue, it sort of worked.
Restoring from S3 is expensive £27.53 for ~400GB
The Scenario
A week or so ago I realised that my hard drive was on the way out, its been on for almost 27,000 hours according to the SMART data. I first noticed when the PC was loading into check disk after every reboot. It took me about 3 reboots to decide something was up and I used Crystal Disk Mark to check the disk and sure enough it was reporting ‘Bad’. So I ordered 2*6TB drives and thought I’d better have a go at moving the data and making sure my backups were up to date.
For my backups, I use cloudberry backup (now called something else) which is an encryptable cloud backup solution which is compatible with Amazon’s S3. I use the cheapest storage option, S3 Glacier Deep Archive.
DDRESCUE GUI Data RecoveryDDRESCUEVIEW Preview
I booted in to a persistent live Ubuntu 20 environment and installed ddrescue, ddrescueview and ddrescue-gui. I found that the tools worked well but took way to long for the drive, you can see in the remaining time section of ddrescue-gui it would have taken an estimated 60 days to recover the data at the fastest setting.
Making DDRESCUE Faster
To make ddrescue faster I found it was best to watch the drive speed in ddrescue-gui and then I scrapped it over the command line for a faster experience.
In the end I used these commands, make sure to replace the drives with your setup and the minimum read rate to one your drive is comfortable with. For the first command, I stopped it at around 90 percent of the way through the drive and swapped it for the second one.
# First run to cover myself in case the drive died more seriously.
sudo ddrescue -f --reopen-on-error --min-read-rate=8000 /dev/sdd2 /dev/sdc1 /home/ubuntu/Documents/log1.log
# Lots of Passes to try to recover slow sections.
sudo ddrescue -f --reopen-on-error --retry-passes=5 /dev/sdd2 /dev/sdc1 /home/ubuntu/Documents/log1.log
Although this really seems like a your mileage may vary environment depending on the type of failure your drive has.
If you do end up using ddrescue-gui at least to begin with, you can use the log file to get you a command to start off with. Make sure to read the manual pages for ddrescue to determine the best command for you.
Here is an example of one of my outputs (.log files),
You can of course view this data using ddrescueview.
ddrescueview
DDRESCUE Results
After a week and a bit, I decided to stop the experiment and see what had been recovered. ddrescueview looked like this,
ddrescueview results
ddrescue was able to recover about 90.83% of the ntfs partition, enough to mount the drive and view the data. It contained many of my important personal files and more importantly images and home video. The actual used space on the drive was only ~700GB, with around ~450GB of data that was valuable to me.
When I opened the personal photos and videos, I found the results to be quite poor, there were glitches in them, sometimes files had no actual data in them, sometimes they had stripes and lines in the image, because of the spread of the failure across the partition blocks, the data was basically a really poor copy with a lot of holes.
ddrescue non-tried sectors made many files unreadable or poor quality
I decided that it was best not to continue the recovery with ddrescue and instead restore from backup, the age of the backup was exactly 1 month prior to the failure to the day, so no real loss. However only the data that I truly cared about was backed up. So stuff like my VMware ISO files and downloads folder were lost and unrecoverable.
Downloading from AWS S3 Glacier Deep Archive
Using Cloudberry I made a restore plan and recovered the data using the slowest SLA at 3-5 days, which by sods law took the full amount of time to process and then some, because I put in the wrong decryption password and needed to re-request the data.
Anyway here is the bill, £27.53
AWS S3 $37.31 (£27.53) Bill
The killer was the data transfer fees out of London, at a cost of $0.09/GB ($28.37).
And with that, all of my data was re-recovered, this time without corruption.
Learnings and Final Thoughts
Although AWS S3 is a valid backup option, its expensive to recover from. I already pay roughly $1/mo for ~400GB (315GB compressed). For larger recovery this would be prohibitively expensive, multi-terabyte or whole disk backups would require compression.
Physical damage to a hard drive is essentially game over, your data is lost. For best results have redundancy. This is the only reason I am thankful for S3, it was my only solution to recover my data. A local backup would have been much cheaper and faster to recover.
The two new 6TB drives run in a Windows storage spaces two-way mirror pool.
Microsoft SQL Server is an extensive and complicated database host, there are many ways of keeping services online during patches or instance failure.
Typically systems that can scale to maintain uptime are regarded as Highly Available. Highly Available systems are designed to be online as much as possible, often with no single points of failure.
SQL Server has many availability technologies built in to achieve robust and consistent SQL Server continuity of service,
Replication
Log Shipping
Mirroring
AlwaysON Availability Groups
SQL Server Big Data Clusters
SQL Server Replication
SQL Server Replication is best suited for site to site connections to remote locations where connectivity is not ideal. Replication has roles for SQL Server, publishers, distributors and subscribers.
SQL Server Replication is best suited for modes of use where data is published to endpoints for dissemination, such as point of sale terminals or batch processing.
SQL Server Replication has not been modified significantly in SQL Server 2016 or SQL Server 2017 and has been available since SQL Server 2008.
SQL Server Log Shipping
SQL Server Log shipping is best suited for architectures where delays of 15 or 30 minute intervals are acceptable for changes in the data to be reflected, such as report generation or cataloguing through analysis tools like Microsoft PowerBi or SQL Server Integration Services for data transformation.
Because Log Shipping involves exporting a set of transactions to network attached storage and importing it into subscriber databases there is inherent delays in changes to the data and databases can fall out of synchronisation.
SQL Server Log Shipping involves configuring the ‘primary’ database to run a log shipping configuration and provide a network location to allow the client to read the transactions from and run restore jobs. When using SSMS this process is more or less straightforward as it will handle creating Jobs on the servers and seeding the database on the downstream secondary databases, as well as providing a standard report on transaction log shipping status to diagnose issues with the configuration if necessary.
SQL Server Log Shipping although easy to set up I have found when out of synchronisation restoring the database to catch up again with the transaction log is not always straightforward compared to other methodologies, especially with large databases.
Log shipping does not failover gracefully out of the box, clients will not be able to natively switch over if the ‘primary’ database has failed.
Mirroring
Using Mirroring in SQL Server is inadvisable as it is soon to be removed in future versions.
Database mirroring relies on having two different SQL Servers mirror each other and the principal (primary) database performs the transactions and the mirror server (secondary) then follows as soon as possible, depending on the configuration this can be asynchronous or synchronous.
Database Mirroring is effectively a more robust version of log shipping with automatic failover and does not require a witness to failover gracefully.
Always On Availability Groups
Always on availability groups can be configured to use Windows fail-over clustering or Linux Pacemaker.
Always on is the preferred option for most configurations due to its relatively easy to use nature and flexibility. Always On Availability Groups rely on a listener that applications connect to and WSFC (Windows Server Failover Cluster) (or Pacemaker) manages the cluster environment to ensure the databases remain up when the primary is unavailable and handle the mediation to elect a new primary. WSFC also allows you to select a methodology for defining a new primary database through quorum.
