判断从AppStore升级XCode是否卡住

从AppStore升级XCode有的时候耗时很长,让你不太清楚到底是卡住了,还是只是很慢。下面的步骤可以帮助你判断:

  1. 保持App Store打开
  2. 打开Console App (注意:不是Terminal)
  3. 在Console App的搜索框里,输入”App Store”
  4. 在Console App里点击”Start”

Log里会记录类似如下信息:

ASDAppQuery.resultsDidChange 497799835: <NSProgress: 0x600003ff8d00> : Parent: 0x0 (portion: 0) / Fraction completed: 0.8230 / Completed: 823 of 1000
ASDAppQuery.resultsDidChange 497799835: <NSProgress: 0x600003fe5500> : Parent: 0x0 (portion: 0) / Fraction completed: 0.8280 / Completed: 828 of 1000

表示此时安装正在正常进行。

如果信任第三方工具,可以用下面这个图形化的工具来取代AppStore以安装XCode:

brew install xcodes

MapStruct with Dagger Injection

jsr330componentModel 配合CONSTRUCTORInjectionStrategy可以与Dagger配合使用。

定义Mapper,并在map的同时完成解密:

package net.axqd.mapper;

import org.mapstruct.InjectionStrategy;
import org.mapstruct.Mapper;
import org.mapstruct.Mapping;

@Mapper(componentModel = "jsr330",
    uses = DecryptorQualifier.class
    injectionStrategy = InjectionStrategy.CONSTRUCTOR)
public interface AToBMapper {
    @Mapping(
        source = "src.encrypted",
        target = "decrypted",
        qualifiedByName = { "DecryptorQualifier", "decrypt" })
    B mapAToB(final src)
}

其中,DecryptorQualifier的定义如下:

package net.axqd.mapper;

import lombok.NonNull;
import org.mapstruct.Named;
import javax.inject.Inject;

@Named("DecryptorQualifier")
public class DecryptorQualifier {
    private final Service service;

    @Inject
    public DecryptorQualifier(@NonNull final Service service) {
        this.service = service;
    }

    @Named("decrypt")
    public String decrypt(final String encrypted) {
        return service.decrypt(encrypted);
    }
}

定义Mapper的Provider:

package net.axqd.dagger;

import net.axqd.mapper.AToBMapper;
import net.axqd.mapper.AToBMapperImpl;

import dagger.Module;
import dagger.Provides;
import javax.inject.Singleton;

@Module
public class MapperModule {
    @Provides
    @Singleton
    AToBMapper aToBMapper(final DecryptorQualifier qualifier) {
        return new AToBMapperImpl(qualifier);
    }
}

定义Component:

import dagger.Component;
import javax.inject.Singleton;

@Component(modules = {
    MapperModule.class
})
@Singleton
public interface FooBarComponent {
    ...
}

单元测试:

package net.axqd.mapper;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;

@ExtendWith(MockitoExtension.class)
public class AToBMapperTest {
    private static final String ENCRYPTED = "encrypted";
    private static final String DECRYPTED = "decrypted";

    @Mock
    private DecryptorQualifier qualifier;

    @InjectMocks
    private AToBMapperImpl mapper;

    @Test
    public void aToBMapper_when_xxx() {
        when(qualifier.decrypt(ENCRYPTED)).thenReturn(DECRYPTED);

        final A a = A.builder().xxx.build();
        final B b = mapper.mapAToB(a);

        assertEquals(...);
    }
}

Private CA for mTLS with cert-manager

第一步,安装cert-manager:

https://cert-manager.io/docs/installation/kubernetes/

第二步,安装自签名ClusterIssuer(所有Namespace共用):

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: FooBarSelfSign
spec:
  selfSigned: {}

第三步,签发CA证书(Application Namespace自用):

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
 name: FooBarCACertificate
spec:
 secretName: FooBarCACertificate
 isCA: true
 issuerRef:
   name: FooBarSelfSign
   kind: ClusterIssuer
 commonName: "FooBar CA"

第四步,安装CA(Application Namespace自用):

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
 name: FooBarCA
spec:
 ca:
   secretName: FooBarCACertificate

第五步,生成随机JKS密码(Application Namespace自用):

NOTE:如果不需要JKS可以省略这步。

apiVersion: v1
kind: Secret
metadata:
  name: FooBarJKSPassword
data:
  Password: {{ uuidv4 | b64enc }}

第六步,使用CA签发Application证书(Application Namespace自用):

NOTE:如果不需要JKS,可以省略keystores的部分

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: FooBarServerCertificate
spec:
  secretName: FooBarServerCertificate
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  keySize: 2048
  keyAlgorithm: rsa
  keyEncoding: pkcs1
  commonName: server.foobar.com
  keystores:
    jks:
      create: true
      passwordSecretRef:
        name: FooBarJKSPassword
        key: Password
  issuerRef:
    name: FooBarCA
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: FooBarClientCertificate
spec:
  secretName: FooBarClientCertificate
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  keySize: 2048
  keyAlgorithm: rsa
  keyEncoding: pkcs1
  commonName: "FooBar Client"
  keystores:
    jks:
      create: true
      passwordSecretRef:
        name: FooBarJKSPassword
        key: Password
  issuerRef:
    name: FooBarCA
    kind: Issuer
    group: cert-manager.io

生成的FooBarServerCertificateFooBarClientCertificate包含以下内容:

  • tls.key – 证书私钥
  • tls.crt – 证书
  • ca.crt – CA证书
  • keystore.jks
  • truststore.jks

第七步,安装签发的证书到应用:

示例1 – Server Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: FooBarServer
spec:
  template:
    spec:
      containers:
        - name: FooBarServer
          volumeMounts:
            - name: tls
              mountPath: /var/ssl/private
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: FooBarServerCertificate
            items:
              - key: truststore.jks
                path: foobar-truststore.jks
              - key: keystore.jks
                path: foobar-keystore.jks
...

示例2 – Ingress Client:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: FooBarIngress
  annotations:
    kubernetes.io/ingress.class: nginx

    # enable mTLS for frontend
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: 2
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false"
    # read the ca.crt part of the certificate
    nginx.ingress.kubernetes.io/auth-tls-secret: <another certificate for frontend>
    
    # enable mTLS for backend
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-secret: FooBarClientCertificate
spec:
  rules:
    - host: server.foobar.com
      http:
        paths:
          - path: /xxx
            backend:
              serviceName: FooBarAppService
              servicePort: xxx
  tls:
    - hosts:
        - server.foobar.com
      # read the tls.key and tls.crt parts of the certificate
      secretName: <another certificate for frontend>
...

Base Template in Helm

有时我们需要使用Base Template来同时生成两个稍微有点差别的manifest文件。

如果你只需要更改Map里面的内容,一种很容易的方式是直接合并生成好的YAML。但对于Containers以及Volumes这种数组类型,合并YAML就不是很容易了。

另外一种方式,是生成本地修改的.Values,具体内容如下:

templates/_util.tpl:

{{- /*
chart.util.merge will merge a YAML value file with context,
use it to generate another YAML file and output the result.
This takes an array of three values:
- the top context
- the name of the value overrides
- the name of the target template
*/ -}}
{{- define "chart.util.merge" -}}
{{- $top := first . -}}
{{- $overrides := fromYaml (include (index . 1) $top) | default (dict ) -}}
{{- include (index . 2) (merge $overrides $top) -}}
{{- end -}}

templates/_parent.yaml:

{{/*
Parent Base Template
*/}}
{{- define "chart.parent.tpl" -}}
Current Value: {{ .Values.test.key.value }}
{{- end -}}
{{- define "chart.parent" -}}
{{- include "chart.util.merge" (append . "chart.parent.tpl") -}}
{{- end -}}

templates/child1.yaml:

{{- include "chart.parent" (list . "chart.parent.child1") -}}
{{- define "chart.parent.child1" -}}
Values:
  test:
    key:
      value: child1
{{- end -}}

templates/child2.yaml:

{{- include "chart.parent" (list . "chart.parent.child2") -}}
{{- define "chart.parent.child2" -}}
Values:
  test:
    key:
      value: child2
{{- end -}}

values.yaml:

test:
  key: {}
    # This value is provided by child1.yaml or child2.yaml
    #value: parent

最后结果如下:

$ helm template test test
---
# Source: test/templates/child1.yaml
Current Value: child1
---
# Source: test/templates/child2.yaml
Current Value: child2

使用 MingW-W64 编译支持的 Linux 项目

下载并安装MSys2和MingW-W64:

  • https://www.msys2.org/
  • http://mingw-w64.org/doku.php/download/mingw-builds/

准备环境:

  • (msys2) # pacman -Syu
  • (close terminal window)
  • (msys2) # pacman -Su
  • (msys2) # pacman -S diffutils (for cmp)
  • (msys2) # pacman -S make

编译:

  • (msys2)# export PATH=/c/mingw-w64/x86_64-8.1.0-win32-seh-rt_v6-rev0/mingw64/bin:$PATH
  • (msys2)# ./configure –host=i686-w64-mingw32
  • (msys2)# make && make install

Jenkins及子任务全自动部署

Ansible基础配置

https://galaxy.ansible.com/geerlingguy/jenkins

Ansible额外配置 – Groovy Scripts

“{{ jenkins_home }}/init.groovy.d/{{ item }}”

Jenkins Seed Job – Job-DSL Groovy Scripts

  • 动态API参考文档: http://<your jenkins>/plugin/job-dsl/api-viewer/index.html
  • Playground: http://job-dsl.herokuapp.com/
  • Deprecated NoTriggerBranchProperty for multibranchPipelineJob
    • 临时configure解决方案
  configure { node ->
    def src = node / sources / data / 'jenkins.branch.BranchSource'
    src << 'buildStrategies' {
      'jenkins.branch.buildstrategies.basic.NamedBranchBuildStrategyImpl' {
        filters {
          'jenkins.branch.buildstrategies.basic.NamedBranchBuildStrategyImpl_-WildcardsNameFilter' {
            includes("")
            excludes("*")
            caseSensitive(false)
          }
        }
      }
    }
  }

Global Shared Libraries – Groovy Scripts

  • http://<your jenkins>/pipeline-syntax/

Jenkinsfile – Groovy Scripts

  • @Library(‘PipelineSharedLib@master’) _

Perforce双Client Checkout解决方案

确保两个client的名称不同:

  • pipelineJob / definition / cpsScm / scm / perforce / workspace / streamSpec with pipelineJob / definition / cpsScm / lightweight and scriptPath
  • checkout() in Jenkinsfile

并从scm中提取相关信息:

  checkout([
    $class:        'PerforceScm',
    credential:    scm.credential,
    workspace:     new StreamWorkspaceImpl(
                     'none',                   // charset
                     false,                    // pinHost
                     scm.workspace.streamName, // streamName
                     "jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" // format
                   ),
    populate:      scm.populate
  ])

快速重建git p4 clone过的仓库

背景

  • Bitbucket/Stash仓库 https://stash.abc.com/scm/test/foo_bar.git
  • Perforce仓库 //Foo/bar
  • Bitbucket/Stash仓库的master分支保持和Perforce的内容完全一致

首次创建并同步

  • $ git p4 clone //Foo/bar
  • $ git remote add origin https://stash.abc.com/scm/test/foo_bar.git
  • $ git push -u origin master
  • $ git checkout -b p4/master
  • $ git push -u origin p4/master

快速重建之前的仓库 (跳过git p4 clone,而直接syncFromOrigin + git p4 rebase)

  • $ git clone https://stash.abc.com/scm/test/foo_bar.git
  • $ git config –add git-p4.syncFromOrigin “true” <– 先读取remotes/origin/p4/master
  • $ git fetch origin
  • $ git checkout master
  • $ git p4 rebase <– 自动重建remotes/p4/master
  • $ git push –atomic origin master

维护remotes/origin/p4/master分支

  • 这个分支中的内容会从Bitbucket/Stash同步,而避免了从Perforce缓慢的提取过程
  • 定期从master分支更新内容到这个分支,可以使得尽可能多的内容从Bitbucket/Stash同步

Win 7的CBS_Persist_X.log占据大量硬盘空间 (76GB in my case)

公司电脑Win7的系统,硬盘空间时不时报警,一直没怎么管,最近实在看不下去了,弄了一下。

  1. 首先,需要找到什么文件占据大量的磁盘空间,推荐一个工具 — WinDirStat
    https://windirstat.net/
  2. 发现C:\Windows\Logs\CBS\目录下有大量的cbs_persist_xxx.log
  3. 原因是Win7一个Bug,微软一直知晓,却一直也没动静:
    http://www.infoworld.com/article/3112358/microsoft-windows/windows-7-log-file-compression-bug-can-fill-up-your-hard-drive.html
  4. 解决方法如下:
    1. 运行->services.msc
    2. 停掉Windows Modules Installer服务
    3. 删除C:\Windows\Logs\CBS\下面的全部内容
    4. 【可选】删除全部C:\Windows\Temp\cab*
    5. 重启电脑

嗯…

手动解密微软Agile Encryption的ECMA-376文档

先给一些参考资料,如果有漏掉的部分,可以参考这里:

  1. [MS-OFFCRYPTO]: Office Document Cryptography Structure
  2. [MS-CFB]: Compound File Binary File Format
  3. Standard ECMA-376 Office Open XML File Formats
  4. Apache POI Encryption Support

一 解析CFB文件结构

先介绍一下CFB文件结构,以从中取出解密需要的信息:

  1. CFB文件被切分为等长的Sector,然后用如下方式组织起来:
    Header Sector + Sector #0 + Sector #1 + …
    由于有Header的存在,算Sector偏移的时候,Sector Number需要加一。
  2. Sector之间由FAT表关联,形成多条Sector链
    FAT[0] – Sector #0 的下一个Sector Number (4 bytes)
    FAT[1] – Sector #1 的下一个Sector Number

    还有一些特殊的内容,比如:
    FAT Sector (0xFFFFFFFD)
    链结尾 (0xFFFFFFFE)
    空的占位符 (0xFFFFFFFF)

    例子:
    FAT[0]: fd ff ff ff
    FAT[1]: 04 00 00 00

  3. FAT表也存储在Sector里,FAT表由DIFAT表关联,关联方式和FAT一致,唯一的区别是前109个DIFAT项,被直接存储在Header Sector里。

    例子:
    DIFAT[0]: 00 00 00 00
    DIFAT[1]: 07 00 00 00

  4. Sector内存储了各种类型的数据,包括一个简单的类似文件系统的树状结构。

    其中,Storage类似于文件夹,Stream类似于文件。
    Root Storage是唯一的根目录,下面挂其他的Storage或者Stream。

    由于Stream大小比较大,还提供了Mini Stream,用于存储比较小的数据。

  5. Header的结构参考[MS-CFB] 2.2 Compound File Header,一些关键信息如下:
    1. Byte Order – 确定字节序 (0xFFFE)
    2. Sector Shift – 单个Sector的大小 0x9 (512B) 或者 0xc (4096B)
    3. Mini Sector Shift – 单个Mini Sector的大小 0x6 (64B)
    4. Mini Stream Cutoff Size – 小于这个大小的数据,被放在Mini Stream里 (4096B)
    5. First Directory Sector Location – Directory Stream的起始Sector Number
    6. First Mini FAT Sector Location – Mini FAT表的起始Sector Number
    7. First DIFAT Sector – DIFAT表的起始Sector Number (如果Header里的109项已经够用了,则为链结尾 – 0xFFFFFFFE)
  6. 接下来解析Directory Stream所在的起始Sector
    1. 偏移:[(Sector Number + 1) * Sector Shift]
    2. 每个Directory Entry的大小是128B,如果Sector大小为512B,则每个Sector可以放四个Directory Entry
    3. Directory Entry的结构参考[MS-CFB] 2.6.1 Compound File Directory Entry,一些关键信息解释如下:
      1. Directory Entry Name – 项名称 一般第一个为Root Entry [UTF-16]
      2. Object Type – 0x0 未分配 0x1 Storage 0x2 Stream 0x5 Root Storage
      3. Child ID:子项Directory Entry的ID (如果没有子项,则为0xFFFFFFFF)
      4. Left Sibling ID: 左兄弟项Directory Entry的ID (如果没有左兄弟项,则为0xFFFFFFFF)
      5. Right Sibling ID: 右兄弟项Directory Entry的ID (如果没有右兄弟项,则为0xFFFFFFFF)
      6. Starting Sector Location – 对Stream而言,表示起始Sector Number;对Root Storage而言,则指示了Mini Stream的起始Sector Number
      7. Stream Size: 对Stream而言,表示数据大小;对Root Storage而言,则表示Mini Stream的大小
    4. 例子
      Directory Entry [0]:
      Directory Entry Name: 52 00 6f 00 6f 00 74 00 20 00 45 00 6e 00 74 00 72 00 79 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [UTF-16]
      Directory Entry Name: Root Entry [UTF-16]
      Directory Entry Name Length: 16 00
      Object Type: 05
      Color Flag: 00 [0x00 Red 0x01 Black]
      Left Sibling ID: ff ff ff ff
      Right Sibling ID: ff ff ff ff
      Child ID: 0a 00 00 00
      CLSID: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      State Bits: 00 00 00 00
      Creation Time: 00 00 00 00 00 00 00 00
      Modified Time: 00 3d 2a 3d 8e 12 d2 01
      Starting Sector Location: 03 00 00 00
      Stream Size: 00 08 00 00 00 00 00 00
  7. 根据以上的内容,我们可以得到两个关键的Stream
    1. EncryptionInfo Stream,这个Stream包含一个明文的XML字符串,包含我们需要的解密相关信息
      1. 从提供的密码 和 EncryptedKeyValue里解密中间密钥 encryption/keyEncryptors/keyEncryptor/encryptedKey
        1. spinCount – 加密多少轮,例如10000
        2. saltSize
        3. blockSize
        4. keyBits – 决定AES128 或者 AES256
        5. hashSize
        6. cipherAlgorithm – 例如AES
        7. cipherChaining – 例如CBC
        8. hashAlgorithm – 例如SHA512
        9. saltValue – 用于加密中间密钥的盐值 (base64 编码)
        10. encryptedKeyValue – 加密后的中间密钥 (base64 编码)
      2. 用上面获得的中间密钥解密实际数据 encryption/keyData
        1. saltSize
        2. blockSize
        3. keyBits
        4. hashSize
        5. cipherAlgorithm
        6. cipherChaining
        7. hashAlgorithm
        8. saltValue
    2. EncryptedPackage Stream,这个Stream包含我们待解密的数据
      1. Stream Size : 8 bytes (无符号整数)
      2. Encrypted Data: variable size

二 解密中间密钥

我们用AES256 + SHA512的组合举例, AES256需要Key和初始向量IV

  1. 预先处理
    1. Salt Value 和 Encrypted Key Value 在XML里是base64编码的,我们需要先解码为二进制 base64.b64decode()
    2. 用户输入的密码也需要用utf-16小尾编码 “password”.encode(“utf-16le”)
  2. 获得Key
    1. SHA512编码 (encryptedKey/saltValue + 用户输入的密码)
      pwHash = hashlib.sha512()
      pwHash.update(saltValue)
      pwHash.update(password)
      key = pwHash.digest()
    2. SHA512编码spinCount轮 Hn = H(count + Hn-1)
      其中count为从0开始的无符号32位数
      for i in xrange(spinCount):
      pwHash = hashlib.sha512()
      pwHash.update(struct.pack(“<I”, i))
      pwHash.update(key)
      key = pwHash.digest()
    3. SHA512编码 Hfinal = H(Hn + BlockKey)
      如果Hfinal大于keyBits,则需要按keyBits截断;
      反之,如果小于,则需要用0x36来append补足
      pwHash = hashlib.sha512()
      pwHash.update(key)
      pwHash.update(struct.pack(‘<BBBBBBBB’, 0x14, 0x6e, 0x0b, 0xe7, 0xab, 0xac, 0xd0, 0xd6))
      key = pwHash.digest()[:32]
  3. 获得IV
    1. 就是encryptedKey/saltValue
    2. 如果大于blockSize,则按blockSize截断
    3. 反之,如果小于,则需要用0x36来append补足
  4. 解密encryptedKey/encryptedKeyValue得到中间密钥
    aes = AES.new(key, AES.MODE_CBC, iv)
    secretKey = aes.decrypt(encryptedKeyValue)

三 解密EncryptedPackage Stream

解密的时候,每4096B为一个解密单元,count从0开始的32位无符号整数。

  1. 获得IV H = H(keyData/saltValue + count)
    contentHash = hashlib.sha512()
    contentHash.update(saltValue)
    contentHash.update(struct.pack(“<I”, count))
    iv = contentHash.digest()
  2. 解密当前单元
    aes = AES.new(secretKey, AES.MODE_CBC, iv)
    content = aes.decrypt(encryptedContent)