Architecture, grid and cloud
Enterprise Architecture
Enterprise Architecture examples
- NAF - NATO Architecture Framework
- NL NORA - NL overheid architectuur
Architecture of Open Source
- AOSAbook - the architecture of open source - great stuff
Enterprise Service Bus ESB
Context brokers
Tools
Grid
Service Oriented Architecture (SOA), Microservices Architecture (MSA) and Service Mesh patterns
SOA and MSA
On the surface, Microservices and SOA are similar. The architecture consists of a set of services. However ...
- SOA Manifesto- Ali Arsanjani, Grady Booch, ...
- SOA- Wikipedia
- SOA can be seen as part of the continuum which ranges from the older concept of distributed computing[6][9] and modular programming, through SOA, and on to current practices of mashups, SaaS, and cloud computing (which some see as the offspring of SOA)
- There are no industry standards relating to the exact composition of a service-oriented architecture, although many industry sources have published their own principles
- Microservices - Wikipedia
- MSA is SOA without the baggage of web service specifications (WS‑*) and an Enterprise Service Bus (ESB), favoring simpler, lightweight protocols
such as REST, rather than WS‑*.
- MSA avoid using ESBs and instead implement ESB‑like functionality in the microservices themselves
- Each service has its own database, a service can use a type of database that is best suited to its needs (polyglot persistence)
- Eclipse Foundation has published a specification for developing microservices, Eclipse MicroProfile
- Microservices.io - Chris Richardson
Service Mesh
- Service Mesh- Wikipedia
- Each service instance is paired with an instance of a reverse proxy server, called a service proxy, sidecar proxy, or sidecar
- Service instance and sidecar proxy share a container, and the containers are managed by a container orchestration tool such as Kubernetes
- Service proxies are responsible for communication with other service instances and can support capabilities
such as service (instance) discovery, load balancing, authentication and authorization, secure communications etc
- Service instances + their sidecars are said to make up the data plane, which includes not only data management but also request processing and response
- There is also a control plane for managing the interaction between services, mediated by their sidecar proxies
- Options include Istio (a joint project among Google, IBM, and Lyft), Linkerd (CNCF project led by Buoyant) etc
Virtualisation
Virtual Machines
Virtual machines (VMs) are an abstraction of physical hardware turning one server into many servers.
The hypervisor allows multiple VMs to run on a single machine.
Each VM includes a full copy of an operating system, the application, necessary binaries and libraries.
Platforms
Containers are an abstraction at the app layer that packages code and dependencies together.
Multiple containers can run on the same machine and share the OS kernel with other containers, each running as isolated processes in user space.
- Docker - container management
- uses OS-level virtualization to deliver software in packages called containers
- what is a container (on Docker.com)
- docker itself is written in Go, originally it used LinuX Containers (LXC) later switched to runC (libcontainer), which runs in the same operating system as its host (allows it to share a lot of host OS resources)
Uses a layered filesystem (AuFS) and manages networking
- containers are isolated from one another and bundle their own software, libraries and configuration files
- communication via bridge, overlay, host, macvlan or additional network plugins, the default is via enabling the default bridge
- networking distinguishes between networking for standalone containers and networking for a swarm
- CNM - Container Networking Model
- main components:
- SaaS version consists of server and client:
- dockerd is a persistent process that manages Docker containers and handles container objects, listens for requests
sent via the Docker Engine API - the names engine and dockerd seem to be used interchangeable
- docker (client) provides CLI to interact with daemons
- Docker objects are entities used to assemble an application in Docker (containers (environment that runs applications), services (scaling, creation of swarm), and images (template to build containers))
- Docker registry is a repository for Docker images, clients download ("pull") images for use or upload ("push") images that they have built
- Swarm: turns a pool of Docker hosts into a virtual, single host
- Kubernetes - k8s - management of containerized applications
- open-source container-orchestration system for automating application deployment, scaling, and management
- designed by Google, maintained by the Cloud Native Computing Foundation
- works with a range of container tools, including Docker
- Vagrant - provisions working environments on top of VirtualBox, VMware, AWS, etc
using provisioning tools such as shell scripts, Chef, or Puppet
- Ansible -open-source software provisioning, configuration management, and application-deployment tool (sponsored by Red Hat)
Cloud
Open source
- OpenDev - a space for collaborative Open Source software development
- Open Stack - began in 2010 as a joint project of Rackspace Hosting and NASA.
As of 2012, it is managed by the OpenStack Foundation, a non-profit corporate entity established in September 2012,
components include
- Compute (Nova)
- Networking (Neutron)
- Block storage (Cinder)
- Identity (Keystone)
- Image (Glance)
- Object storage (Swift)
- Dashboard (Horizon)
- Orchestration (Heat)
- Workflow (Mistral)
- Telemetry (Ceilometer)
- Database (Trove)
- Elastic map reduce (Sahara)
- Bare metal (Ironic)
- Messaging (Zaqar)
- Shared file system (Manila)
- DNS (Designate)
- Search (Searchlight)
- Key manager (Barbican) - https://wiki.openstack.org/wiki/Barbican
- Container orchestration (Magnum)
- Root Cause Analysis (Vitrage)
- Rule-based alarm actions (Aodh)
Commercial
- Terraform - by HashiCorp - multicloud deployment
- Terraform Wikipedia
- Amazone AWS/EC2 - webservices elastic cloud
- AWS security services
- AWS Identity & Access Management - Manage User Access and Encryption Keys
- Amazon Cognito - Identity Management for your Apps
- Amazon GuardDuty - Managed Threat Detection Service
- Amazon Inspector - Analyze Application Security
- Amazon Macie - Discover, Classify, and Protect your Data
- AWS Artifact - On-demand access to AWS compliance reports
- AWS Certificate Manager - Provision, Manage, and Deploy SSL/TLS Certificates
- AWS CloudHSM - Hardware-based Key Storage for Regulatory Compliance
- AWS Directory Service- Host and Manage Active Directory
- AWS Firewall Manager - Central Management of Firewall Rules
- AWS Resource Access Manager - Simple, secure service to share AWS resources
- AWS Secrets Manager - Rotate, Manage, and Retrieve Secrets
- AWS Security Hub - Unified security and compliance center
- AWS Shield - DDoS Protection
- AWS Single Sign-On - Cloud Single Sign-On (SSO) Service
- AWS WAF - Filter Malicious Web Traffic
- Mozilla on AWS security
- AWS Perspective - build network diagrams of your cloud
- AWS cryptographic services
- CloudHSM provides HSMs to store a variety of cryptographic keys, including master keys and data keys
- Key Management Service (KMS) provides tools for generating master and data keys, interacts with other AWS services to encrypt their data
- Encryption SDK provides a client-side encryption library for implementing encryption and decryption operations on all types of data
- DynamoDB Encryption Client provides a client-side encryption library for encrypting data tables before sending them to a database service,
such as Amazon DynamoDB
- Microsoft Azure
- Gluster - GlusterFS
- scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace
- POSIX compatible, accessible via NFS and SMB
- GlusterFS is a userspace filesystem (getting modules into linux kernel is a long and difficult process).
Being a userspace filesystem, to interact with kernel VFS, GlusterFS makes use of FUSE (File System in Userspace).
- Supports five types of volumes:
- Distributed Glusterfs Volume
- Replicated Glusterfs Volume
- Distributed Replicated Glusterfs Volume
- Striped Glusterfs Volume
- Distributed Striped Glusterfs Volume
- inter-node connection protected by built-in firewall
- Gluster SSL for TLS authentication
- Gluster ACL Access Control Lists
- Google Cloud
- Google OpenCloud
- Cloudstack- Apache
Less commercial
Cloud security