Skip to main content

OpSec

路 One min read

Two-Factor Authentication (2FA)#

Enable Two-Factor Authentication on every service, and keep your device with you.

FileVault#

Enable FileVault to encrypt disk

Ref

warning

Lock Your Desktop When You're Away.

Seperate browsers for different purposes#

For example, I usually used two browsers(Chrome Stable and Chrome Canary). Chrome Stable for my personal use, I will install any Chrome plugin I want on this browser. Chrome Canary for company use, without any Chrome plugin installed.

run-multiple-versions-of-chrome-side-by.html

Email#

Register SPF for your domain to detect forging sender addresses

SPF

Privacy#

Whenever we are using the software, some of them will collect your data. For example, search history.

Privacy tool

Periodically check your data not leaked#

https://haveibeenpwned.com/

GCP resource hierarchy

路 2 min read

Overview#

Scenario:

  • A company size of 200 people seperated into 10 teams.
  • Administer can have the company-wide policy to limit GCP account.
  • Easy way to switch user to the other team.
  • Single login page for user to login
  • Each team wants its own access to its resources.
  • Also, different environment for specific purpose.
  • Sometimes, they need temporarily cooperation between teams.
  • For auditing, bills should be seperated by each team.

Solution:

  • Google Group to grouping user by team or functionality.
  • Nesting group to organize members within the team.
  • GCP folder to have hiearchy architecture.
  • Set IAM role by Google Group email
  • Set Organization Policy on GCP folder limit resource.
  • Using seperated GCP folder for different enviroment, producion, staging...... .

Organization#

graph LR R[Organization root] --> A[Team A folder] R[Organization root] --> B[Team B folder] A[Company folder] --> A1[Team A production folder] A[Company folder] --> A2[Team A staging folder] B[Company folder] --> B1[Team B production folder] B[Company folder] --> B2[Team B staging folder] R[Organization root] --> P[playground folder]

Nested Google Group#

graph LR subgraph A [Team A Google Group] A1[Team A production Google Group] A2[Team A staging Google Group] end

IAM relationship between Google Group and Google project#

graph LR A[Team A Google Group] -- Folder Admin/Owner -->AF[Team A folder] A1[Team A production Google Group] -- Folder Admin/Owner -->A1F[Team A production folder] A2[Team A production Google Group] -- Folder Admin/Owner -->A2F[Team A staging folder] P[playground Google Group] -- Folder Admin/Owner -->PF[Team B playground folder]

Terraform#

Create service account per GCP project as Terraform repo's credential.

graph LR; A[A GCP project] --> AT[A Terraform repo]; AT[A Terraform repo] --> A[A GCP project]; B[B GCP project] --> BT[B Terraform repo]; BT[B Terraform repo] --> B[B GCP project];

KVM GPU passthrough Ubuntu 20.04

路 4 min read

Environment#

  • Host: Ubuntu 20.04
  • Guest: Ubuntu 20.04
  • GPU: NVIDIA庐 GeForce庐 RTX 2080 Ti

Enable IOMMU#

Configure GRUB#

Edit /etc/default/grub

# Intel CPUGRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"# AMD CPUGRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1"

Update GRUB#

sudo update-grub

Reboot#

sudo shutdown -r now

Verify IOMMU is enabled#

dmesg | grep IOMMU

Output:

IOMMU enabled

Enable IOMMU group#

Check IOMMU group is enabled#

for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort

output:

/sys/kernel/iommu_groups/0/devices/0000:00:00.0/sys/kernel/iommu_groups/1/devices/0000:00:04.0/sys/kernel/iommu_groups/2/devices/0000:00:04.1/sys/kernel/iommu_groups/3/devices/0000:00:04.2/sys/kernel/iommu_groups/4/devices/0000:00:04.3

Edit BIOS setting is not enabled#

If output is not expected, configure BIOS setting

VT-d#

(Asus)

  • Advanced => System Agent Configuration => Intel VT for Directed I/O (VT-d)

(Supermicro)

Isolation of the guest GPU#

graph LR subgraph C [guest] C1[PCI device] end subgraph B [hypervisor] B1[VFIO] --> C1[PCI device] end subgraph A [Host] A1[PCI device] --> B1[VFIO] end

Using vfio-pci to manage PCI device#

Find out vendor ID and device ID#

lspci -nn | grep -i NVIDIA

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)01:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:1ad6] (rev a1)01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:1ad7] (rev a1)

GeForce RTX 2080 Ti VGA compatible controller: PCI ID:01:00.0 vendor ID: 10de device ID: 1e04

Configure GRUB#

/etc/default/grub

Apply all the related devices

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on vfio-pci.ids=10de:1e04,10de:10f7,10de:1ad6,10de:1ad7"

Update GRUB#

sudo update-grub

Reboot#

sudo reboot

Verify PCI device is managed by vfio-pci#

lspci -nnv

Find the line Kernel driver in use

0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1) (prog-if 00 [VGA controller])...Kernel driver in use: vfio-pci

Test GPU passthrough on kvm instance#

Fresh install#

Run virt-install with --host-device [device_id] and --features kvm_hidden=on parameters

virt-install ... \--host-device 01:00.0 \--features kvm_hidden=on \

Modify existing instance#

virsh edit [domain]

Add PCI mapping hostdev block

0000:01:00.0 within the host will be mapped to 0000:04:00.0 within guest

warning

bus number should less than virtio's

Increase virtio's bus number to spare small number for new added entry

<devices>  ...    <hostdev mode='subsystem' type='pci' managed='yes'>      <source>        <address domain='0x0000' bus='0x01' slot='0x00'function='0x0'/>      </source>      <address type='pci' domain='0x0000' bus='0x04'slot='0x00' function='0x0'/>    </hostdev>    <memballoon model='virtio'>      <address type='pci' domain='0x0000' bus='0x05'slot='0x00' function='0x0'/>    </memballoon>    <rng model='virtio'>      <backend model='random'>/dev/urandom</backend>      <address type='pci' domain='0x0000' bus='0x06'slot='0x00' function='0x0'/>    </rng></devices>

kvm hidden within features block

<features>...  <kvm>    <hidden state='on'/>  </kvm></features>

Check GPU is working in guest#

lspci

04:00.0 VGA compatible controller: NVIDIA Corporation TU102[GeForce RTX 2080 Ti] (rev a1)

Install NVIDIA driver

sudo apt updatesudo apt install nvidia-driver-460sudo reboot

nvidia-smi

Wed Mar 10 08:19:43 2021+---------------------------------------------------------------------------+| NVIDIA-SMI 460.39       Driver Version: 460.39       CUDVersion: 11.2     ||-------------------------------+--------------------+----------------------+| GPU  Name        Persistence-M| Bus-Id        Disp.A Volatile Uncorr. ECC || Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage GPU-Util  Compute M. ||                               |                    |               MIG M. ||===============================+====================+======================||   0  GeForce RTX 208...  Off  | 00000000:04:00.0 Of|                  N/A || 15%   44C    P0     1W / 250W |      0MiB / 11019Mi|      0%      Default ||                               |                    |                  N/A |+-------------------------------+--------------------+----------------------++---------------------------------------------------------------------------+Processes:                                                                ||  GPU   GI   CI        PID   Type   Procesname                  GPU Memory ||        ID ID                                                 Usage      |===========================================================================||  No running processefound                                                 |+---------------------------------------------------------------------------+

Reference#

AWS resource hierarchy

路 3 min read

Overview#

Scenario:

  • A company size of 200 people seperated into 10 teams.
  • Administer can have the company-wide policy to limit AWS account.
  • Easy way to switch user to the other team.
  • Single login page for user to login
  • Each team wants its own access to its resources.
  • Also, different environment for specific purpose.
  • Sometimes, they need temporarily cooperation between teams.
  • For auditing, bills should be seperated by each team.

Solution:

Organization#

graph LR R[Organization root] --> C[Company OU] C[Company OU] --> A1[Team A production AWS account] C[Company OU] --> A2[Team A staging AWS account] C[Company OU] --> B1[Team B production AWS account] C[Company OU] --> B2[Team B staging AWS account] R[Organization root] --> P[playground AWS account]

Company OU will whitelist AWS resource and region. Playground account has no limitation on AWS resource. Periodically clean resource of playground account.

info

AWS Tag is another solution to have more detail bill within AWS account.

Terraform#

Create terraform IAM access key and attach AdministratorAccess policy as Terraform repo's credential.

graph LR; A[A AWS account] --> AT[A Terraform repo]; AT[A Terraform repo] --> A[A AWS account]; B[B AWS account] --> BT[B Terraform repo]; BT[B Terraform repo] --> B[B AWS account];

SSO#

caution

Recommended using SSO to manage AWS Accounts, otherwise you will have to manage your account from different place.

Google apps SSO#

warning

You have to have GSuite super administrator access.

Follow the How to Set Up Federated Single Sign-On to AWS Using Google Apps instruction

Then, you will find out it's one-to-one mapping between Google account and IAM role.

We want mapping of Google group and IAM role.

Refer repo https://github.com/1Strategy/sso-to-aws-using-gsuite

By using GSuite Admin API

Iterate the Google group, for each Google account, map to corresponding IAM role

AWS SSO#

https://aws.amazon.com/tw/single-sign-on/

graph LR; A[A AWS account admin group] --> AA[Administer role of A account]; A[A AWS account readonly group] --> AR[readonly role of account]; B[B AWS account admin group] --> BA[Administer role of A account]; B[B AWS account readonly group] --> BR[readonly role of account];

Assign user to group to have access to account.

Neat short-term credentials for command line interface