Cloud Storage, Databases and Data Services

A storage service must fit the way an application reads and changes data. A database that updates a few records, a team editing shared documents and a website serving photos have different needs.

Three ways to organize data

A block volume contains fixed-size blocks. The operating system formats it with a file system, just as it would format a physical disk. It can update a small part of a file in place. This suits a database making many small reads and writes. A volume normally serves one virtual machine; sharing one directly requires disk support and software designed for clustered access.

File storage supplies a shared folder over a network. NFS is common on Linux; SMB is common on Windows. Several clients can open the same share, and file locking coordinates access. An existing application that expects a shared path can keep that pattern without being rewritten to call an object API.

What an object contains

An object has its data, a unique key and metadata, such as its content type. In a flat-namespace bucket, invoices/2026/10/inv-1042.pdf is one complete key. The slashes group names by prefix; they do not create ordinary disk folders.

Applications commonly upload objects with PUT requests and download them with GET requests over HTTPS. General-purpose object access treats data as an object: replacing its content typically means uploading a replacement, while a block disk supports small in-place writes. Some object services add specialized append or directory features. A bucket still has a different access model from a shared file server.

Object storage can hold images, recordings, logs and backups in a collection that grows without provisioning a disk for each increase. Standard online classes serve active websites and analytics with millisecond access. Delayed retrieval belongs to particular archive classes, not to object storage as a whole.

Choose storage for an application

A web shop uses a block disk for its database server because records change frequently. Five web servers read common page templates from an NFS file share. Product photos and 2 TB of nightly backups go into object storage. The same application can use all three types.

Temporary host storage, such as AWS instance store, the Azure temporary disk and Google Cloud Local SSD, suits scratch files and caches that can be rebuilt. Host failure or VM lifecycle events can erase it. Keep the only copy of important data on persistent storage, which exists independently of a running VM.

Pricing follows the storage model. Object storage generally bills actual stored GB plus requests and transfers. Block storage bills provisioned capacity, including unused space. File storage may bill used space, as EFS does, or reserved capacity. Start a choice with the access pattern, then compare the bill for that pattern.

Object storage separates the data from the server that uses it. A bucket or container holds the collection, while each object has its own key. Location, access and recovery settings determine how safely that collection can be used.

Match the provider's names

An Azure storage account can also hold file shares, queues and tables. S3 general purpose bucket names, Cloud Storage bucket names and Azure storage account names use shared namespaces, so another customer's name may be unavailable. The name also forms part of the object's address. For example, an Azure blob can have the address https://shopmedia.blob.core.windows.net/images/products/lamp-01.jpg.

An S3 bucket belongs to an AWS Region. Azure storage accounts have a region and redundancy choice. Cloud Storage offers region, dual-region and multi-region locations. Location affects latency, price and data residency, meaning the area where the data is kept.

Share without exposing the collection

Keep the collection private and grant the access the application needs through identity permissions and resource policies. For a temporary download, use an S3 presigned URL, an Azure shared access signature (SAS) or a Cloud Storage signed URL. An invoice link that expires in 60 minutes gives temporary access without making the whole bucket public. Anyone holding a usable signed link can exercise its permissions during that period.

Stored objects are encrypted by default with provider-managed keys; customer-managed keys give the customer more control. HTTPS protects data during transfer. Encryption and permissions solve different problems: someone with valid read permission can still read an encrypted object through the service.

Recover, retain and upload

Versioning keeps earlier object versions so that an overwrite can be undone. Stored versions add to the bill. Azure and Google Cloud also offer soft delete, which retains deleted data for a recovery period. Immutable storage, or write once, read many (WORM), blocks changes and deletion during retention. Examples are S3 Object Lock, Azure immutable blob storage and Cloud Storage retention policies with bucket lock.

Large uploads can be split into parts, uploaded in parallel and retried individually. S3 calls this multipart upload; Azure block uploads and Google composite uploads provide related approaches. A dropped connection need not restart the whole large transfer.

Size a growing collection

One million photos averaging 3 MB occupy 3,000,000 MB, or 3,000 GB: 3 TB in decimal units. Another 20,000 photos a day add 60 GB daily. Over 365 days, that is another 21.9 TB. Object storage accommodates this growth without reserving a fixed disk in advance.

In flat namespaces, moving a folder-like prefix involves changing object keys. Hierarchical options add actual directory operations, including Azure Data Lake Storage and Cloud Storage hierarchical namespace. Check the namespace before treating a displayed folder as a directory.

After choosing a storage type, match its access method and performance to a service. A high-performance disk and a shared folder can both store files, but they meet different application needs.

Disks and their copies

Disk types offer different performance. Hard-disk-based volumes suit large sequential reads; SSDs suit frequent small operations. IOPS means input/output operations per second. Choose both capacity and performance, because a large disk is not necessarily the right disk for a busy database.

An EBS volume belongs to one Availability Zone and attaches to instances there. Azure managed disks are locally redundant by default, with a zone-redundant option. Google offers zonal disks and regional disks replicated across two zones. These location choices set the failures a disk can withstand.

A snapshot captures a volume at a point in time. EBS and Google standard snapshots are incremental: after the first copy, later snapshots store changed blocks. Azure offers full and incremental snapshots. If a volume contains 200 GB and 5 GB changes, an incremental snapshot stores about 5 GB of new blocks. Restoring that snapshot still reconstructs the complete volume, not just the changed portion. It can become a new disk in another zone; snapshots can also be copied across regions.

Shared files and compatible software

EFS is an automatically growing NFS file system for Linux servers, including servers across Availability Zones. FSx offers Windows File Server, Lustre, NetApp ONTAP and OpenZFS. FSx for Windows File Server supplies SMB and Active Directory integration; Lustre suits large, fast data sets in high-performance computing.

Azure Files offers SMB and NFS shares for compatible Windows, Linux and macOS clients. Azure NetApp Files serves demanding enterprise file workloads. Filestore supplies managed NFS on Google Cloud; NetApp Volumes provides NetApp-based shares. Match the protocol and required file-system features before comparing capacity prices.

Connect a local file server

AWS Storage Gateway is a local virtual appliance connecting familiar file, disk or tape interfaces to AWS storage. It caches frequently used data locally. Tape Gateway lets existing backup software use virtual tapes. Azure File Sync synchronizes Windows file servers with Azure Files; cloud tiering keeps hot files locally and fetches colder files when opened.

For a branch with 8 TB of shared files and a 2 TB local disk, tiering can keep the frequently used 2 TB nearby while the cloud holds the full set. Opening a cold file then depends on the network connection.

Deleting a VM does not always delete its data disks; an unattached disk can still incur charges. Snapshot protection also depends on location and account isolation. EBS and Azure snapshots remain in the source region unless copied. Google standard snapshots default to the nearest multi-region. An isolated recovery copy protects against a compromised source account.

A storage tier changes what you pay to keep and retrieve objects. Hot tiers charge more for storage and less for access. Colder tiers lower the storage rate but can add retrieval charges, minimum durations and restore delays.

Compare classes by access needs

S3 Standard-IA and One Zone-IA have 30-day minimums. Glacier Instant Retrieval and Flexible Retrieval have 90-day minimums; Deep Archive has 180 days. Flexible Retrieval takes minutes to hours to restore. Deep Archive standard retrieval completes within 12 hours, bulk within 48 hours. S3 Express One Zone offers fast access within a single zone.

Azure cool has a 30-day minimum, cold 90 days and archive 180 days. Hot, cool and cold provide millisecond access. Archive must be rehydrated to an online tier, which can take up to 15 hours. A storage account sets a default online tier; individual blobs can have their own tier.

Cloud Storage Standard has no minimum; Nearline has 30 days, Coldline 90 and Archive 365. Google Archive remains online, with millisecond access and retrieval fees. The word "archive" alone does not tell you how long a read will take.

Minimum duration is a billing commitment

Deleting a cool Azure blob after 21 days leaves 9 billable days of its 30-day minimum. Deleting an archive blob after 45 days leaves 135 days of its 180-day minimum. A Cloud Storage Archive object deleted after 100 days leaves 265 days. Moving out of a class early can also trigger this charge.

Work through the total bill

Use example rates of $0.020 per GB-month for hot storage and $0.002 for archive, plus $0.02 per GB retrieved from archive. For 50,000 GB held for 12 months:

• Hot storage costs $12,000.
• Archive storage costs $1,200.
• Reading the entire archive once adds $1,000, making $2,200 total.
• Reading it every month adds $12,000, making $13,200 total.

The cheaper storage rate loses its advantage when reads are frequent. A complete estimate also includes request charges, early deletion and egress, the charge for data leaving the cloud. Internet uploads are generally free; downloads are commonly billed.

Let access patterns drive changes

S3 Intelligent-Tiering automatically moves unused objects after 30 and 90 days and returns them on access; it charges monitoring fees but no retrieval fees. Azure smart tier moves between hot, cool and cold by usage. Cloud Storage Autoclass manages classes for a bucket. For fixed aging schedules, lifecycle rules give explicit control.

Single-zone classes trade failure protection for cost or performance. S3 One Zone-IA suits secondary or rebuildable data because losing that zone can lose the data.

Data often becomes less useful as it ages. A lifecycle policy applies that pattern to each new object, so people do not have to remember every move or deletion.

A rule has three parts

A filter identifies objects, such as keys starting with logs/ or objects with a particular tag. A condition states when the rule applies, such as days since creation or, in Azure, last access. An action changes the storage tier or deletes the object. The services are S3 Lifecycle, Azure Blob Storage lifecycle management and Cloud Storage Object Lifecycle Management.

For application logs needed often for one month and retained for seven years, move to an infrequent-access tier on day 30 and an archive tier on day 90. The infrequent tier holds each log for 60 days, exceeding its 30-day minimum. Set deletion after the full calendar retention period: seven years can include two leap days, so 2,557 days avoids deleting early. A simple 2,555-day rule can fall short.

Count time spent inside each tier

The age of the object and its time in a tier are different. If an object enters a 90-day-minimum tier on day 30 and leaves on day 60, its stay is only 30 days:

It leaves 60 days of the minimum unused, which can be billed. Count the interval between transitions before judging a policy's cost. Each transition also costs a request. Some colder classes have minimum billable object sizes, such as 128 KB in S3 Standard-IA, so tiny objects may save little or cost more when moved.

Versions and protected retention

Lifecycle rules can remove noncurrent versions after a chosen interval, keep a limited number of earlier versions, and clean up incomplete multipart uploads. Without those rules, versions and abandoned parts can keep consuming paid storage.

An active retention lock or legal hold prevents lifecycle deletion of protected data. For invoices kept for ten calendar years, one plan moves them to archive after one year and schedules deletion after 3,653 days, allowing for up to three leap days. A compliance-mode lock prevents early deletion even by administrators. The lifecycle rule manages aging; the lock enforces protection.

Retrieval is a separate operation

Azure high-priority rehydration usually takes under one hour for blobs below 10 GB, at a higher price. Azure lifecycle rules cannot rehydrate an archived blob; restoring it needs a separate request. Plan a restore before promising an urgent delivery from an offline archive.

Lifecycle actions run in the background, commonly within a day or two of the condition. Use the policy to apply retention consistently across the collection, and allow for that processing time when planning operations.

Data can survive an outage and still be unreachable during it. Storage protection therefore has two separate measures: durability and availability.

Read the percentages correctly

Durability describes protection against data loss over a year. S3 Standard is designed for 99.999999999% durability, called eleven nines. The loss fraction is one in 100 billion per object per year. For 10 million objects, the expected annual loss is 0.0001 objects: an average of one every 10,000 years. This is a statistical design measure, not a schedule for losing an object.

Availability describes the share of time data can be reached. For a 30-day month there are 43,200 minutes. At 99.9% availability, the unavailable share is 0.1%, or 43.2 minutes. At 99.99%, it is 4.32 minutes; at 99%, 432 minutes, or 7.2 hours. S3 Standard's design availability is 99.99%. An SLA is a contractual commitment, often backed by credits; its figure can differ from the design target.

Match copies to the failure

Several copies inside one data center protect against hardware failure. Copies across Availability Zones protect against a data-center outage. Copies in a second region address a regional disaster, subject to where and how replication occurs.

Synchronous replication confirms a write after the required copies acknowledge it. Some systems require all designated copies; others use a majority, or quorum. Asynchronous replication confirms at the primary and sends changes onward afterward. If the primary is lost, changes still in transit may be lost. That replication lag must stay within the business's RPO, its maximum acceptable amount of recent data loss measured in time.

Decode Azure redundancy

LRS offers at least eleven nines durability, ZRS twelve and geo-redundant options sixteen. GRS has six copies in total. Ordinary GRS/GZRS secondary data is unavailable for reads until failover; RA-GRS and RA-GZRS add read access to the secondary. A replicated region is not necessarily an immediately active replacement.

Most S3 classes span at least three zones; One Zone classes do not. S3 replication can copy objects to another bucket in the same or a different Region. Cloud Storage regional buckets span zones, dual-region buckets use two chosen regions, and multi-region buckets use a wider area. Optional turbo replication targets copying new data within 15 minutes in dual-region storage.

Data residency applies to secondary copies too. For a one-country requirement, check the second region's location. Replication can also carry unwanted overwrites or deletions, depending on the service and settings. It supplies resilient copies of current state; protected versions and backups supply recovery from a bad state.

A backup is a recoverable copy kept for deletion, corruption, attack or site loss. A useful backup plan starts with what the business can lose and how long it can remain offline.

Set two separate targets

A backup every 24 hours can leave almost 24 hours of changes unprotected if failure happens just before the next backup. It meets a 24-hour RPO, but not a 1-hour RPO. Hourly backups or sufficiently current continuous replication address the shorter target.

Recovery speed is independent. If restoring 3 TB takes 6 hours and the RTO is 4 hours, the plan misses the target even if the backup contains every required record. A faster restore process or a warm standby can reduce recovery time.

Apply policies across resources

AWS Backup centralizes backup policies for resources including EBS, RDS, DynamoDB, EFS, FSx and S3. Azure Backup covers resources including VMs, managed disks, Azure Files, blobs and SQL Server on VMs. Google Cloud offers Backup and DR Service. Policies set frequency, retention and copies to other regions or accounts.

Managed databases also take backups and retain transaction logs. RDS, Azure SQL Database and Cloud SQL support point-in-time restore to a selected moment within retention. RDS retention can be up to 35 days. Logs allow recovery between backup times, such as immediately before an accidental update.

Isolate a recovery copy

The 3-2-1 rule guides backup planning: three copies, on two kinds of storage, with one off-site. In the cloud, another region helps with site loss, and a separate account helps isolate access. Locked retention adds protection against an attacker deleting the backup: AWS Backup Vault Lock, Azure immutable backup vaults and Google backup vaults with enforced retention provide this control.

A daily off-site copy does not meet a four-hour RPO for a regional disaster. It can lag by almost a day even when local snapshots run every four hours.

For a 3 TB file share with a four-hour RPO and eight-hour RTO, take four-hourly snapshots for local recovery and make a recoverable cross-region copy at least every four hours. Keep recent snapshots for seven days and daily recovery history for 90 days. Test a complete restore each quarter and measure whether it finishes within eight hours.

Recovery services and archives

AWS Elastic Disaster Recovery and Azure Site Recovery replicate servers so they can start elsewhere with short recovery targets, often minutes, at extra cost. They address bringing a working system back, not just preserving old files.

An archive keeps inactive records for long-term use or retention. A backup supports recovery of a current system with rolling history. A low-cost archive can be useful storage for copies, but its retrieval delay must fit the recovery plan. Restore tests check both whether the data works and how long recovery really takes.

A relational database stores related facts in tables. A managed service runs the database platform for you, while your team remains responsible for how its data and queries work.

Read tables and keys

A table contains rows for records and columns for attributes. Its schema defines those columns and their data types. A primary key uniquely identifies a row. A foreign key refers to another table's primary key and connects records across tables.

Suppose customers contains Ana with customer ID 1 and country Brazil, and Kofi with ID 2 and country Ghana. The orders table contains orders 501 and 503 for customer 2, with totals of 80 and 45, and order 502 for customer 1, totaling 120. SQL reads and changes the data:

SELECT name FROM customers WHERE country = 'Ghana';

This returns Kofi. Joining on customer_id connects Kofi to orders 501 and 503, whose total is 125. The customer name does not have to be repeated in every order.

Keep a transaction together

A transaction groups changes into one unit. The ACID properties describe its guarantees:

• Atomicity: all steps succeed, or none do.
• Consistency: committed data obeys its defined rules.
• Isolation: concurrent transactions are kept from interfering in ways the isolation level disallows.
• Durability: a committed change survives a crash.

A transfer of 50 debits one account and credits another. If the second change fails, atomicity undoes the first too. This is why relational databases suit orders, inventory and payments: connected changes can commit together.

Choose who runs the platform

A managed PostgreSQL service gives a new application automated platform maintenance. A legacy database needing an old engine version and an operating-system plug-in may need a VM. Managed service does not mean the provider chooses your schema, fixes slow queries or grants the right users access. Your team also selects size, region and availability settings.

Distinguish extra copies

A standby replica in another zone supports failover if the primary fails. Examples include RDS Multi-AZ, zone-redundant Azure SQL Database and Cloud SQL high availability. Failover can briefly interrupt service; it is not zero downtime.

A read replica spreads read traffic and may be placed in another region. It commonly receives changes asynchronously, so it can show an older value just after a write. It is not the same as a failover standby or a historical backup.

Vertical scaling adds processors or memory, often with a short interruption. Read replicas scale read traffic, while automatic storage growth adds disk capacity. Backups and transaction logs support point-in-time restore after a harmful change that replicas may already have copied.

Begin a database choice with the application: its engine, required features and scale. Keeping the same engine often avoids code changes. A compatible cloud engine can offer different storage and scaling behavior without being identical to the original.

Compare the main services

Amazon RDS supports MySQL, PostgreSQL, MariaDB, Oracle, SQL Server and IBM Db2. Aurora is a separate AWS engine compatible with MySQL or PostgreSQL. It stores six copies across three Availability Zones, grows storage automatically and offers Aurora Serverless for changing compute demand.

Azure SQL Database uses the SQL Server engine and offers serverless and Hyperscale options. SQL Managed Instance supports almost all instance-level SQL Server behavior, useful for existing databases that depend on features such as SQL Server Agent jobs. Azure Database for PostgreSQL and Azure Database for MySQL run their open-source engines as managed flexible servers. SQL Server on Azure Virtual Machines supplies full operating-system control, with customer-managed maintenance.

Cloud SQL runs MySQL, PostgreSQL and SQL Server. AlloyDB for PostgreSQL provides a PostgreSQL-compatible service optimized for demanding workloads, including analytical queries. Spanner distributes relational data globally, scales horizontally and keeps strong consistency across regions. Multi-region configurations offer an SLA up to 99.999% availability.

Choose by the feature that matters

A regional MySQL retailer can use RDS for MySQL, Azure Database for MySQL or Cloud SQL for MySQL. A SQL Server application depending on instance-level jobs needs an instance-compatible service, such as SQL Managed Instance; verify specific feature support for any target. A demanding PostgreSQL application may consider Aurora PostgreSQL or AlloyDB.

Global replication is not one single behavior. Aurora Global Database has one primary write Region and asynchronous secondary Regions for local reads and disaster recovery. Spanner addresses globally distributed, strongly consistent transactions. A small single-region application usually has no need for that global complexity.

If the workload is unpredictable or often idle, a serverless option can adjust capacity. If the application requires OS access, select a VM. "Managed" and "serverless" describe operational choices; they do not make every engine feature interchangeable.

Include licenses and usage in the bill

Commercial engines such as Oracle, SQL Server and Db2 require an included license or a valid bring-your-own-license arrangement, called BYOL. Azure Hybrid Benefit can reuse eligible SQL Server licenses. Open-source engines such as MySQL and PostgreSQL have no engine license fee.

Typical charges include compute, GB-months of storage, extra backup storage, data transferred out and sometimes I/O operations. Serverless charges follow capacity used over time. Compare the complete service and license arrangement, not just the hourly compute figure.

NoSQL databases organize data around particular access patterns. "Not only SQL" covers several models, not one replacement for every relational database. Choose a model by the questions the application asks and the guarantees its answers need.

Match the model to the question

A key-value store quickly retrieves cart:58213 by its known key. It has fewer ways to search arbitrary attributes. A document groups related fields, and documents can differ in shape:

{"orderId":"A-1042","customer":{"name":"Ana"},"items":[{"sku":"LAMP-01","qty":2},{"sku":"BULB-07","qty":4}]}

The application can read this order and its items together. The flexible shape does not remove structure: the application still needs usable fields and carefully chosen keys. Some services also enforce validation rules.

Wide-column stores handle huge write volumes, such as readings from millions of sensors. Graph stores make relationships explicit as edges between nodes. Tracing connections among accounts is a graph problem, while repeatedly finding one account by ID is a key lookup.

Use memory for repeated reads

An in-memory store keeps data in RAM for very fast access, often below a millisecond. A common pattern is cache-aside: check the cache; on a miss, read the database and cache a copy with an expiry time.

For a 1 ms cache lookup and a 50 ms database query, with a 90% hit rate, a hit costs 1 ms and a miss costs 51 ms. The average is:

The miss includes the failed cache lookup. Cached data may expire or be evicted, so an ordinary cache should not hold the only authoritative copy. Durable in-memory database services are a distinct choice.

Choose consistency deliberately

With strong consistency, a read returns the latest confirmed write. Eventual consistency allows a replica briefly to show an older value while updates catch up. This can support lower latency, lower cost or better availability during some failures. The required behavior depends on the task: a just-updated balance and a recommendation counter may have different needs.

DynamoDB offers eventually and strongly consistent reads for supported operations. Cosmos DB offers five consistency levels. NoSQL does not mean "no transactions": many services support transactions, though the supported scope differs.

Relational databases suit rich relationships, joins and multi-row transactions. NoSQL suits very large scale, flexible records and known access patterns. An application can keep orders in a relational store, profiles in a document store and repeated product reads in a cache.

Cloud NoSQL services package different models and operating choices. First identify whether the workload needs documents, connections, wide-column data or a cache, then compare compatibility and capacity charges.

Recognize the service families

The table groups services by purpose; it does not mean they have identical APIs. DynamoDB is serverless, supports key-value and document access, and offers global tables across Regions. DocumentDB targets MongoDB compatibility. Keyspaces is Cassandra-compatible, and Neptune stores graphs.

Cosmos DB is globally distributed and offers NoSQL, MongoDB, Cassandra, Gremlin and Table APIs. An account uses one API. It also offers five consistency levels. Azure Table Storage is a separate, simple key-value service within a storage account. Sharing a Table interface does not give it all Cosmos DB's global distribution and consistency features.

Firestore is a serverless document database with real-time updates and offline device support. Bigtable handles large, low-latency wide-column workloads and supports the HBase API. Sensor histories and personalization at very large scale are common uses.

Caching and durable memory are different

ElastiCache supports Valkey, Redis OSS and Memcached. Memorystore offers Valkey, Redis and Memcached, while Azure Managed Redis provides managed Redis caching. Frequently repeated product queries can use these caches to avoid repeated database reads.

Amazon MemoryDB is designed as a durable in-memory database. An ordinary cache can evict or lose an entry; MemoryDB's durable database role is different. Decide whether the data is a replaceable copy before choosing solely by response time.

Pay for reserved or actual throughput

With provisioned capacity, reserve throughput in advance. DynamoDB uses read and write capacity units; Cosmos DB uses request units per second, or RU/s. Predictable, steady demand can make this economical. With on-demand/serverless capacity, charges follow requests and capacity adjusts to demand. Storage is billed separately.

A new shopping cart with unpredictable spikes can start with DynamoDB on-demand or Cosmos DB serverless. A mobile document app that must work offline fits Firestore's device features. A fraud team tracing links between accounts needs a graph service, not merely a faster key lookup.

The separate Cosmos DB for PostgreSQL service is being retired; it is not another account API. Microsoft directs new distributed PostgreSQL projects to Elastic Clusters in Azure Database for PostgreSQL. Match the actual service and API before assuming an existing application's compatibility.

A large upload is a calculation before it is a tool choice. Work out how much network capacity the move can actually use, then compare that time with the complete offline transfer process.

Put size and speed in matching units

Storage uses bytes; networks use bits per second. One byte contains 8 bits. With decimal units, 1 GB is one billion bytes, 1 TB is 1,000 GB and 1 Gbps is one billion bits per second. A binary GiB is about 1.074 GB, so check which unit a tool displays.

For 100 TB over 1 Gbps, the data contains 800 trillion bits. Dividing by one billion gives 800,000 seconds, about 9.3 days. At 80% effective use, divide by 0.8 Gbps: 1,000,000 seconds, about 11.6 days. At 100 Mbps, even full use takes about 93 days. A 1 Gbps link moves at most 125 MB/s, or about 10.8 TB daily, not 1 GB/s.

Match online tools to the interface

DataSync includes scheduling, encryption and integrity checks and can move supported file-system or object sources into S3, EFS or FSx. Transfer Family suits partners already sending files through established transfer protocols. A one-time copy and continuing synchronization are different tasks.

Compare with offline transfer

Azure Data Box devices provide up to 120 TB or 525 TB, with Data Box Disk for smaller sets. Google Transfer Appliance also moves data on shipped hardware. Include ordering, copying, shipping and provider upload time; the full process can take one to a few weeks. Encrypt the device and verify the resulting data.

AWS Snowball Edge is closed to new customers and its commercial-Region support ends December 31, 2026. Current AWS routes include DataSync and Data Transfer Terminal, a reserved physical location where customers bring drives for high-speed uploads.

A site has a 200 Mbps link, with half available for migration. The effective rate is 100 Mbps. Moving 300 TB takes 24 million seconds, about 278 days, making an offline device attractive. Moving 2 TB takes 160,000 seconds, about 1.9 days, so shipping would add delay.

Repeated transfers need a continuing path

AWS Direct Connect, Azure ExpressRoute and Google Cloud Interconnect provide dedicated private connectivity with more consistent speed and latency. They can suit large ongoing transfers. An offline device seeds a copy but does not keep later changes synchronized.

Include transfer-service charges and later egress in the estimate. Internet ingress is generally free; cloud downloads commonly cost per GB. Use encryption in transit and checksums or equivalent integrity checks to confirm that the destination copy is complete.

Moving database bytes is only part of migration. The target must understand the schema, the application must work with it, and the final changes must arrive before users switch over.

Classify what changes

A homogeneous migration keeps the engine, such as PostgreSQL to managed PostgreSQL. A heterogeneous migration changes it, such as Oracle to PostgreSQL. That can require converting data types, schemas, stored procedures and application code.

A same-engine move can still need version or feature checks. A conversion report is useful precisely because it identifies what does not transfer automatically.

Limit the downtime

An offline migration stops the application for export, transfer and import. An online migration begins with a full load while the source remains in use. Change data capture (CDC) then sends new changes to the target. The initial copy alone is already out of date if users kept writing during it.

Cutover requires a controlled sequence:

1. Stop writes to the source.
2. Apply the remaining captured changes and confirm the target has caught up.
3. Promote the target to an independent database where required.
4. Redirect the application and verify it works.

Continuous replication reduces the final window; it does not eliminate the need for that window or for tests. Keeping the old source read-only for a period can provide a fallback without allowing competing writes.

Choose discovery, conversion and migration tools

AWS DMS moves databases and supports ongoing replication. AWS SCT and DMS Schema Conversion assess and convert database structures and code. Application Discovery Service gathers server information; Application Migration Service rehosts servers. Migration Hub tracks work, and Migration Evaluator supports the business case.

Azure Migrate discovers, assesses and migrates servers, databases and web applications. Azure Database Migration Service handles supported SQL Server, PostgreSQL and MySQL moves, using online or offline methods. Data Box transports large offline data copies; it does not perform schema conversion.

Google Database Migration Service supports moves into Cloud SQL or AlloyDB, including supported continuous migrations and heterogeneous conversions to PostgreSQL. Migration Center discovers assets, assesses them and estimates costs. Select a supported source-to-target path, not merely a service with "migration" in its name.

Work through a changing-engine move

A 24/7 shop moves a 2 TB Oracle database to a PostgreSQL-compatible managed target. Assess conversion first. Create the target and convert the schema, rewrite the procedures flagged for manual work, and start full load plus CDC. Test the application while replication continues. Then perform the controlled cutover. A folder of exported files in a bucket is a useful transfer artifact, but users need a consistent, running target database.

The system that records today's transactions has a different job from the system explaining five years of business history. Analytics combines data from many sources without making routine transactions wait behind large scans.

Recognize data shapes and workloads

Structured data has a fixed layout, such as a sales table. Semi-structured data carries labels without one fixed table shape, such as JSON or XML. Unstructured data includes images, audio and free-form documents.

OLTP, online transaction processing, handles many small reads and writes of current records. OLAP, online analytical processing, runs large queries over history. Heavy reports on a production database can slow orders and payments, so analytics commonly uses a separate copy.

Compare lake, warehouse and lakehouse

A warehouse combines sources into consistent tables. Its columnar storage can read only needed columns. If a 2 TB table has 100 equally sized columns and a query needs three, the approximate scan is 60 GB, not 2 TB.

A lake preserves raw files for exploration, data science and machine learning, including data whose later use is not known. Without a catalog and ownership, it becomes hard to find or trust anything: a "data swamp." Lakehouses add warehouse-like management with formats such as Apache Iceberg and Delta Lake. A lake and warehouse can coexist because they serve different stages and needs.

Follow a pipeline

A data pipeline ingests data, stores it, transforms it, analyzes it and presents or acts on results. With ETL, extraction is followed by transformation and then loading. With ELT, data is loaded first and transformed in the destination. Cloud warehouses can supply the processing capacity for ELT.

Batch processing handles a collected period, such as a night's sales load. Streaming handles events as they arrive, suitable for live dashboards, suspicious payments or sensor alerts. A historical monthly report may work well in batches; a reaction needed within seconds calls for streaming.

Keep data usable and trustworthy

Governance assigns rules and processes for access, retention, meaning and quality. A catalog records what exists and what it means. Lineage traces where data came from and which transformations changed it. Together they help a team explain a number on a dashboard.

Quality checks address completeness, accuracy, consistency, timeliness, validity and uniqueness. A missing region code hurts completeness; duplicate payments hurt uniqueness; an outdated feed hurts timeliness. Quality is useful before both analysis and AI. Open formats and interoperable tools make sharing easier and reduce isolated data silos and dependence on one vendor.

An analytics service has a specific place in the path from events to decisions. Identify that job before choosing a provider product: capturing a stream, cleaning it, querying it and displaying it are different operations.

Warehouses and queries over files

Redshift offers provisioned and serverless warehouses. BigQuery is serverless, separates storage from compute and offers scan-based or capacity-based query pricing. Microsoft Fabric combines warehouse and lakehouse features; Synapse Analytics remains available, with Fabric the direction for new development.

Athena runs serverless SQL on files in S3; it does not store the data it queries. BigQuery external tables can read Cloud Storage files, and Fabric lakehouse tables in OneLake have a SQL analytics endpoint. Querying files in place is different from loading another copy into a database.

Prepare data and handle events

AWS Glue transforms data and maintains a Data Catalog describing structure and location. Azure Data Factory builds and schedules pipelines. Google Dataflow runs both batch and streaming processing with Apache Beam.

For Spark and related big-data frameworks, use Amazon EMR, Azure Databricks or Fabric Spark, and Google's Managed Service for Apache Spark, formerly Dataproc. These supply processing frameworks, not the same role as a dashboard or a message service.

For streams, Amazon Kinesis captures and processes real-time data, and Data Firehose loads streams into destinations. Azure Event Hubs ingests events; Stream Analytics or Fabric Real-Time Intelligence analyzes them. Google Pub/Sub carries messages, and Dataflow processes them.

Follow one event to the dashboard

For a live shop dashboard, order events enter Kinesis, Event Hubs or Pub/Sub. Raw copies land in S3, Azure Data Lake Storage or Cloud Storage. Glue, Data Factory or Dataflow cleans the records. Redshift, Fabric or BigQuery supports analysis, and Quick Sight, Power BI or Looker shows the results.

Looker can define business metrics centrally so teams use the same meaning for measures such as revenue. Amazon OpenSearch Service has a different role: indexing text and logs for search and analysis.

Reduce scan-based query charges

At an example rate of $5 per TB scanned, reading 2 TB costs $10. If a columnar format such as Parquet lets the query read three of 100 equally sized columns, it reads 60 GB, or 0.06 TB, costing $0.30. If the files are also partitioned by day and the query needs only one of 30 equally sized days, the scan falls to 2 GB, costing $0.01.

The savings come from actually excluding unneeded columns and partitions. Merely storing a file in an object store does not ensure a small scan. BigQuery capacity reservations are a billing option; they do not make the customer administer warehouse servers.

Cloud AI services range from ready-made APIs to platforms for training models. The best starting layer depends on the business task, the data and the team's expertise.

Connect the main ideas

Artificial intelligence covers systems performing tasks such as recognizing images or understanding language. Machine learning is an approach within AI: a model learns patterns from data. Deep learning uses many-layered neural networks. Generative AI creates content such as text, images or code from a prompt. A foundation model is trained broadly for adaptation to many tasks; a large language model, or LLM, focuses on language.

Training produces or adjusts a model from data. Inference uses a trained model on new input. Training a small model may need only an ordinary processor; large deep-learning models often need GPUs or TPUs. Scoring a new purchase for fraud is inference, even if the model was trained months earlier.

Classification predicts a category; regression predicts a number. These methods can simplify complicated rules, forecast demand, personalize recommendations and detect anomalies. Agentic AI adds planning and tool use so an agent can carry out a task, such as searching information and calling an application.

Choose a service layer

1. Pre-trained APIs handle established tasks such as translation or speech recognition without building a model.
2. Generative platforms provide foundation models, grounding and agent tools: Amazon Bedrock, Microsoft Foundry and Gemini Enterprise Agent Platform, formerly Vertex AI.
3. ML platforms build, train and deploy custom models: SageMaker AI, Azure Machine Learning and Gemini Enterprise Agent Platform. AutoML reduces the code needed to train from your data.
4. ML within data services, such as BigQuery ML and Redshift ML, lets teams work through SQL where the data already sits.

A retailer translating descriptions can use a translation API. A chat assistant answering from company manuals can use retrieval-augmented generation (RAG): retrieve relevant manual passages and provide them with each question. A demand forecast based on the retailer's own sales can use AutoML, database ML or an ML platform. Building from scratch adds effort when an existing service already fits.

Evaluate the data and outputs

Missing, outdated or biased training examples can produce poor or unfair results. Check completeness, accuracy, consistency, timeliness, validity and uniqueness. Responsible AI also requires fairness, privacy, security, transparency, explainability, human oversight and accountability.

A confident generative answer can contain a hallucination, a fluent but false claim. Ground important outputs in trusted information and review them. The provider secures its service; the customer controls supplied data, access and how outputs are used. RAG gives evidence to the model, but its presence alone does not guarantee a correct answer.

Many common AI tasks already have a pre-trained service. An application sends input through an API and receives results without its developers first training a model.

Match the input and output

Vision finds objects, scenes and text in images; Google's Video Intelligence API handles video analysis. Speech-to-text transcribes audio, while text-to-speech generates spoken output from written text. Text analysis finds sentiment, entities, key phrases or categories. Translation changes the language of the text.

Document services preserve relationships within forms, such as which date belongs to the label "Invoice date" or which amount is a table total. General image text detection finds text but does not provide the same form understanding. An invoice workflow therefore calls for document extraction, not merely image labels.

Combine services around a business process

A contact center can transcribe recorded calls, analyze the sentiment in each transcript and display recurring unhappy topics on a dashboard. Transcription produces the words; language analysis interprets aspects of those words.

An insurer can extract names, dates and amounts from claims with a document service and send low-confidence results to a person. That review is part of the process: a detected field is a prediction, not a guarantee that the value is correct.

Amazon Lex builds voice or text chatbots and can work with Amazon Connect, a cloud contact center. Amazon Q provides generative assistance for business users and developers. Microsoft Foundry and Gemini Enterprise Agent Platform provide tools for conversational assistants and agents. Azure Vision, Speech, Language and related APIs sit within Foundry Tools, formerly Azure AI services.

Estimate use before estimating price

Services may charge per image, audio minute, page, translated character or generated token. Count the relevant unit, then apply the rate. Use an example translation rate of $15 per million characters. Two thousand descriptions with 500 characters each contain one million characters. Translating into ten languages processes ten million characters and costs $150. Counting source text only would miss the repeated processing for each language.

Pre-trained does not always mean fixed behavior. Custom transcription vocabularies can help with product names; custom image labels can recognize a company's products; custom document classifiers can sort its forms. These options can tailor a service without a complete model-building project.

Images, voices and documents may contain personal data. Supply them for a lawful purpose, obtain consent where required and control access and retention. Choose the service by its task and data needs before deciding that custom training is necessary.