Architecture, grid, booting, emulation, virtualisation, cloud
See also deployment solutions.
Contents
Enterprise Architecture
Basics
Enterprise Architecture examples
- NAF - NATO Architecture Framework
- NL NORA - NL overheid architectuur
Architecture of Open Source
- AOSAbook - the architecture of open source - great stuff
REST
Basics
Technologies (queues, brokers)
Message queues
Enterprise Service Bus ESB
Context brokers
Grid
Service Oriented Architecture (SOA), Microservices Architecture (MSA) and Service Mesh patterns
SOA and MSA
On the surface, Microservices and SOA are similar. The architecture consists of a set of services. However ...
- SOA Manifesto- Ali Arsanjani, Grady Booch, ...
- SOA- Wikipedia
- SOA can be seen as part of the continuum which ranges from the older concept of distributed computing[6][9] and modular programming, through SOA, and on to current practices of mashups, SaaS, and cloud computing (which some see as the offspring of SOA)
- There are no industry standards relating to the exact composition of a service-oriented architecture, although many industry sources have published their own principles
- Microservices - Wikipedia
- MSA is SOA without the baggage of web service specifications (WS‑*) and an Enterprise Service Bus (ESB), favoring simpler, lightweight protocols
such as REST, rather than WS‑*.
- MSA avoid using ESBs and instead implement ESB‑like functionality in the microservices themselves
- Each service has its own database, a service can use a type of database that is best suited to its needs (polyglot persistence)
- Eclipse Foundation has published a specification for developing microservices, Eclipse MicroProfile
- Microservices.io - Chris Richardson
Service Mesh
- Service Mesh- Wikipedia
- Each service instance is paired with an instance of a reverse proxy server, called a service proxy, sidecar proxy, or sidecar
- Service instance and sidecar proxy share a container, and the containers are managed by a container orchestration tool such as Kubernetes
- Service proxies are responsible for communication with other service instances and can support capabilities
such as service (instance) discovery, load balancing, authentication and authorization, secure communications etc
- Service instances + their sidecars are said to make up the data plane, which includes not only data management but also request processing and response
- There is also a control plane for managing the interaction between services, mediated by their sidecar proxies
- Options include Istio (a joint project among Google, IBM, and Lyft), Linkerd (CNCF project led by Buoyant) etc
Firmware and booting
In general
According to Wikipedia
When a computer is turned off, its software — including operating systems, application code, and data—remains stored on non-volatile memory (NVMEM). When the computer is powered on, it typically does not have an operating system or its loader in RAM. The computer first executes a relatively small program stored in read-only memory (ROM, and later EEPROM, NOR flash) along with some needed data, to initialize RAM (especially on x86 systems), to access the nonvolatile device (usually block device, eg NAND flash) or devices from which the operating system programs and data can be loaded into RAM.
- There are two stages distinguished.
- First-stage bootloaders more or less correspond to firmware and include BIOS, UEFI, coreboot, Libreboot, and Das U-Boot.
- Second-stage boot loaders include GNU GRUB, rEFInd, BOOTMGR, Syslinux, NTLDR or iBoot
- They are not themselves operating systems, but are able to load an operating system properly and transfer execution to it; the operating system subsequently initializes itself and may load extra device drivers. The second-stage boot loader does not need drivers for its own operation, but may instead use generic storage access methods provided by system firmware such as the BIOS or Open Firmware, though typically with restricted hardware functionality and lower performance.
See Debian booting for overview (firmware, bootloader, init shellscript, /sbin/init).
Firmware (BIOS/EFI/UEFI, trustworthy firmware)
BIOS (legacy)
Old school.
EFI (legacy)
Intel developed the original Extensible Firmware Interface (EFI) specifications. The Linux kernel has been able to use EFI at boot time since early 2000s, using the elilo EFI boot loader or, more recently, EFI versions of GRUB.
UEFI
The Unified Extensible Firmware Interface (UEFI) defines a software interface between OS and firmware and replaces the Basic Input/Output System (BIOS) firmware interface. UEFI can support remote diagnostics and repair of computers, even with no operating system installed.
UEFI defines a boot manager as part of its specification (it does not rely on boot sectors). When a computer is powered on, the boot manager checks the boot configuration and based on its settings, then executes the specified OS boot loader or operating system kernel.
The boot configuration is defined by variables stored in NVRAM, including variables that indicate the file system paths to OS loaders or OS kernels.
UEFI provides a shell environment, which can be used to execute other UEFI applications, including UEFI boot loaders.
For Debian and UEFI see local Debian info.
Trustworthy firmware
- trustedfirmware.org (Cambridge) - ARM, Google, Linaro, ...
- provides a reference implementation of secure software for Armv8-A, Armv9-A and Armv8-M
- TF-A project - reference implementation of secure world software for Armv7-A and Armv8-A class processors
- TF-A overview by STM explaining TF-A is divided in several binaries. For 32-bit Arm processors (AArch32), the trusted boot is divided into four stages (in order of execution):
- Boot loader stage 1 (BL1) application processor trusted ROM - BL1 is the first stage executed, and is designed to act as ROM code; it is loaded and executed in internal RAM.
- Boot loader stage 2 (BL2) trusted boot firmware - BL2 is in charge of loading the next-stage images (secure and non secure). To achieve this role, BL2 has to initialize all the required peripherals. At the end of its execution, after having loaded BL32 and the next boot stage (BL33), BL2 jumps to BL32.
- Boot loader stage 3-2 (BL32) trusted runtime firmware - BL32 provides runtime secure services. On Armv7 architecture, the BL32 must embed a Secure Monitor as it will be executed in the same privilege level (PL1-SVC Secure)
- Boot loader stage 3-3 (BL33) non-trusted firmware
- BL1 and BL2 are parts of TF-A, BL32 can be either in TF-A or outside (for example OP-TEE).
- BL33 is outside of TF-A. This is the first non-secure code loaded by TF-A. During the boot sequence, this is the secondary stage boot loader (SSBL).
- TF-A can manage its configuration with a device tree. In the BL2 stage, it is a reduced version of the Linux kernel one, with only the required devices used during boot.
- TF-M project - implements the Secure Processing Environment (SPE) for Armv8-M, Armv8.1-M architectures (e.g. the Cortex-M33, Cortex-M23, Cortex-M55, Cortex-M85 processors) or dual-core platforms
- TF-RMM project - Implementation of the Realm Management Monitor (RMM) Specification which is part of Arm Confidential Compute Architecture (Arm CCA). Arm CCA is an architecture which provides Protected Execution Environments called Realms.
- OPTEE project - companion to a non-secure Linux kernel running on Arm; Cortex-A cores using the TrustZone technology. OP-TEE implements TEE Internal Core API v1.1.x which is the API exposed to Trusted Applications and the TEE Client API v1.0, which is the API describing how to communicate with a TEE.
- Trusted Services project - framework for developing and deploying device Root Of Trust (RoT) services across a range of secure processing environments such as those provided by OP-TEE and Hafnium.
- ...
Boot loaders
U-Boot
Das U-Boot is a boot loader.
It can be built for all boards for which there's a configuration in /u-boot/configs, this includes e.g. qemu_arm64_defconfig and a few other qemu_'s. Also a generic linux boot loader can be build as 'sandbox'.
For all supported boards there are ready-to-use default configurations available; just type "make _defconfig". E.g. for a TQM823L module type: $ cd u-boot, $ make TQM823L_defconfig.
can be built natively to run on a Linux host using the 'sandbox' board. This allows feature development which is not board- or architecture-
specific to be undertaken on a native platform. The sandbox is also used to run some of U-Boot's tests.
See doc/arch/sandbox/sandbox.rst for more details.
See local files in /home/marc/OPTEE/u-boot/out/arm/source/board/sandbox
U-Boot intended start-up flux
Should apply for both SPL and U-Boot proper Note: "SPL" stands for "Secondary Program Loader," which is explained later.
Execution typically starts with an architecture-specific (and possibly CPU-specific) start.S file, such as arch/arm/cpu/armv7/start.S
From there, three functions are called:
lowlevel_init():
- purpose: essential init to permit execution to reach board_init_f()
- no global_data or BSS
- there is no stack (ARMv7 may have one but it will soon be removed)
- must not set up SDRAM or use console
- must only do the bare minimum to allow execution to continue to
board_init_f()
- this is almost never needed
- return normally from this function
board_init_f():
- purpose: set up the machine ready for running board_init_r(): i.e. SDRAM and serial UART
- global_data is available
- stack is in SRAM
- BSS is not available, so you cannot use global/static variables, only stack variables and global_data
Non-SPL-specific notes:
- dram_init() is called to set up DRAM. If already done in SPL this
can do nothing
SPL-specific notes:
- you can override the entire board_init_f() function with your own
version as needed.
- preloader_console_init() can be called here in extremis
- should set up SDRAM, and anything needed to make the UART work
- there is no need to clear BSS, it will be done by crt0.S
- for specific scenarios on certain architectures an early BSS *can* be made available (via CONFIG_SPL_EARLY_BSS by moving the clearing of BSS prior to entering board_init_f()) but doing so is discouraged. Instead it is strongly recommended to architect any code changes or additions such to not depend on the availability of BSS during board_init_f() as indicated in other sections of this README to maintain compatibility and consistency across the entire code base.
- must return normally from this function (don't call board_init_r() directly)
board_init_r():
- purpose: main execution, common code
- global_data is available
- SDRAM is available
- BSS is available, all static/global variables can be used
- execution eventually continues to main_loop()
Non-SPL-specific notes: - U-Boot is relocated to the top of memory and is now running from
there.
SPL-specific notes:
- stack is optionally in SDRAM, if CONFIG_SPL_STACK_R is defined and CONFIG_SYS_FSL_HAS_CCI400 ...
The following options need to be configured:
- CPU Type: Define exactly one, e.g. CONFIG_MPC85XX
- Board Type: Define exactly one, e.g. CONFIG_MPC8540ADS
- ... very long list ... TPM, USB, ...
- long list of configuration settings
Building U-Boot
In case of tool chain problems we recommend to use the ELDK (see https://www.denx.de/wiki/DULG/ELDK) which is extensively used to build and test U-Boot.
(1) Set CROSS_COMPILE
If you are not using a native environment, you need GNU cross compiling tools in your path. In this case, you must set the environment variable CROSS_COMPILE in your shell.
Note that no changes to the Makefile or any other source files are necessary.
For example using the ELDK on a 4xx CPU, please enter:
$ CROSS_COMPILE=ppc_4xx-
$ export CROSS_COMPILE
(2) Configure u-boot
After installing the sources you must configure U-Boot for one specific board type. This is done by typing: $ make NAME_defconfig where "NAME_defconfig" is the name of one of the existing configurations; see configs/*_defconfig for supported names.
(3) make all
Finally, type "make all", and you should get some working U-Boot images ready for download to / installation on your system:
- "u-boot.bin" is a raw binary image // found in /home/marc/OPTEE/u-boot/out/arm
- "u-boot" is an image in ELF binary format // found in same dir
- "u-boot.srec" is in Motorola S-Record format // found in same dir
By default the build is performed locally and the objects are saved in the source directory.
? which is where? because the objects are found in /home/marc/OPTEE/u-boot/out/arm
=> so if we do have u-boot.bin available, why does the top level make say
Makefile:1616: *** 'BL33=/home/marc/OPTEE/build/../u-boot/u-boot.bin' was specified, but '/home/marc/OPTEE/build/../u-boot/u-boot.bin' does not exist. Stop.
make[1]: Leaving directory '/home/marc/OPTEE/trusted-firmware-a'
make: *** [Makefile:207: arm-tf] Error 2
> what is BL33 good for?
> what is trusted-firmware-a and its role?
Debian booting
See Debian booting for Debian.
Secure booting
UEFI Secure Boot and Shim
The UEFI 2.3.1 Errata C specification (or higher) defines a protocol known as Secure Boot, which can secure the boot process by preventing the loading of UEFI drivers or OS boot loaders that are not signed with an acceptable digital signature.
Shim is a boot loader to chain-load signed boot loaders under Secure Boot. Shim becomes the root of trust for all the other distro-provided UEFI programs. It embeds a further distro-specific CA key that is itself used for signing further programs (e.g. Linux, GRUB, fwupdate). This allows for a clean delegation of trust - the distros are then responsible for signing the rest of their packages. Shim itself should ideally not need to be updated very often, reducing the workload on the central auditing and CA teams.
Windows 10 allows OEMs to decide whether or not Secure Boot can be managed by users of their x86 systems.
The Machine Owner Key (MOK) allows you to add signed files.
See secure Debian booting for Debian.
TPM
See Debian TPM for a starter.
Emulation
QEMU
For installation of QEMU refer to LTK.
- QEMU - Quick Emulator
- QEMU’s Tiny Code Generator (TCG) allows to emulate a number of CPU architectures on any supported host platform. Both System Emulation and User Mode Emulation are supported depending on the guest architecture.
- QEMU - Wikipedia
- System emulation: emulates a full computer system, including peripherals, can be used to provide virtual hosting of several virtual computers on a single computer. QEMU can boot many guest operating systems, including Linux, Solaris, Microsoft Windows, DOS, and BSD. It supports emulating several instruction sets, including x86, MIPS, 32-bit ARMv7, ARMv8, PowerPC, RISC-V, SPARC, ETRAX CRIS and MicroBlaze.
- User-mode emulation: runs single Linux/Darwin/macOS programs compiled for a different instruction set. System calls are thunked for endianness and for 32/64 bit mismatches. Fast cross-compilation and cross-debugging are the main targets for user-mode emulation.
- Hypervisor Support: either acts as either a Virtual Machine Manager (VMM) or as a device emulation back-end for virtual machines running under a hypervisor. The most common is the Linux's KVM but the project supports a number of hypervisors including Xen, Apple's HVF, Window's WHPX and NetBSD's nvmm.
- QEMU documentation
- QCOW, QEMU 's own virtual disk image format qemu-img command allows to inspect, check, create, convert, resize and take snapshot of qcow images
- QEMU download for Linux : apt-get install qemu-system
For local info refer to LTK.
Virtualisation
A hypervisor (or virtual machine monitor, VMM, virtualizer) is a kind of emulator; it is computer software, firmware or hardware that creates and runs multiple virtual machines (VM). A computer on which a hypervisor runs one or more virtual machines is called a host machine, and each virtual machine is called a guest machine.
Virtual Machines
VM solutions
Virtual machines (VMs) are an abstraction of physical hardware turning one server into many servers. Each VM includes a full copy of an operating system, the application, necessary binaries and libraries.
- IBM VM - the roots of VM
- Virtual Box - by Oracle, x86 and AMD64/Intel64,
- VMware - part of EMC/Dell
- VMware's most notable products are its hypervisors. VMware became well known for its first type 2 hypervisor known as GSX. This product has since evolved into two hypervisor product lines:
VMware's type 1 hypervisors running directly on hardware and their hosted type 2 hypervisors.
- Type 1: ESXi is a bare-metal hypervisor that runs directly on server hardware without requiring an additional underlying operating system.
- Type 2: managed by e.g. the VMware Tools. These services start when the guest operating system starts.
The service passes information between host and guest operating systems.
- Hyper-V - Microsoft
- KVM (for Kernel-based Virtual Machine) - Linux
- virtualisation on x86 hardware containing virtualisation extensions (Intel VT or AMD-V)
- consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko
Linux virt-manager
The virt-manager application is a desktop user interface for managing virtual machines through libvirt. It primarily targets KVM VMs, but also manages Xen and LXC (linux containers). It presents a summary view of running domains, their live performance & resource utilization statistics. Wizards enable the creation of new domains, and configuration & adjustment of a domain’s resource allocation & virtual hardware. An embedded VNC and SPICE client viewer presents a full graphical console to the guest domain.
VM security
In 2016, AMD introduced Secure Encrypted Virtualization (SEV), the first x86 technology designed to isolate virtual machines (VMs) from the hypervisor.
In 2017, AMD introduced the SEV-ES (Encrypted State) feature which added additionalprotection for CPU register state. In SEV-ES, the VM register state is encrypted on each hypervisor transitionso thatthe hypervisor cannotsee the data actively being used by the VM.
SEV-Secure Nested Paging (SNP) adds strong memory integrity protection tohelp prevent malicious hypervisor-based attacks like data replay, memory re-mapping, and more in order to create an isolated execution environment.
- AMD SEV - secure encrypted virtualization
Virtual Disks
Creating a virtual disk image (VDI) of optical media or a hard disk drive is done to make the content available to one or more virtual machines. VMs emulate a CD/DVD drive by reading an ISO image. This can also be faster than reading from the physical optical medium.
A hard disk drive or solid-state drive in a VM is implemented as a disk image, either
- VHD format used by Microsoft's Hyper-V,
- VDI format used by Oracle Corporation's VirtualBox,
- VMDK format used for VMware virtual machines,
- the QCOW format used by QEMU.
Virtual hard disk images tend to be stored as either a collection of files (where each one is typically 2GB in size), or as a single file. Virtual machines treat the image set as a physical drive.
For analysis: use Steampipe or Turbot.
- Amazon security workshops
- AWS Well-Architected - guidance including on security
- MITRE cloud attack framework
- Mozilla on AWS security
- AWS Perspective - build network diagrams of your cloud
- Amazon security including
- Nitro enclaves - TEE
- AWS Identity & Access Management - Manage User Access and Encryption Keys
- AWS Cognito user sign-up, sign-in, and access control to your web and mobile apps
- Main components: user pools and identity pools.
- A user pool is user directory in Amazon Cognito. With a user pool, users can sign in to web/mobile app through Amazon Cognito. Users can also sign in through social identity providers and through SAML identity providers. All members of the user pool have a directory profile that you can access through a Software Development Kit (SDK).
- After successfully authenticating a user, Cognito issues JWTs that you can use to secure and authorize access to your own APIs, or exchange for AWS credentials.
- Identity pools provide AWS credentials to grant users access to other AWS services. An identity pool exchanges user pool tokens for AWS credentials.
- AWS Single Sign-On - Cloud Single Sign-On (SSO) Service
- AWS Directory Service- Host and Manage Active Directory
- AWS Resource Access Manager - Simple, secure service to share AWS resources
- Amazon Inspector - Analyze Application Security
- AWS Macie discover sensitive data in S3
- AWS Security Hub - allows to centralise all security tool results and analysis
- AWS Artifact - On-demand access to AWS compliance reports
- AWS Secrets Manager (for infrastructure/code secrets) - Rotate, Manage, and Retrieve Secrets - through metadata 169.254.169.254
- Secrets Manager enables you to replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
Also, you can configure Secrets Manager to automatically rotate the secret for you according to a specified schedule. This enables you to replace long-term secrets with short-term ones.
- AWS IMDSv2 - Instant Metadata Service
- AWS CloudHSM - Hardware-based Key Storage for Regulatory Compliance
- AWS Certificate Manager - Provision, Manage, and Deploy SSL/TLS Certificates
- AWS Guardduty - Managed Threat Detection Service - effective but not for free
- Alternatives: Microsoft Azure Sentinel (even for AWS), DataDog
- AWS Shield - DDoS Protection
- AWS Firewall Manager - Central Management of Firewall Rules
- AWS WAF - Filter Malicious Web Traffic
- AWS cryptographic services
- CloudHSM provides HSMs to store a variety of cryptographic keys, including master keys and data keys
- Key Management Service (KMS) provides tools for generating master and data keys, interacts with other AWS services to encrypt their data
- Encryption SDK provides a client-side encryption library for implementing encryption and decryption operations on all types of data
- DynamoDB Encryption Client provides a client-side encryption library for encrypting data tables before sending them to a database service,
such as Amazon DynamoDB
MSFT
Google
Less commercial
Cloud security