오답

오답

자꾸 틀리는 문제 정리;

1. Posterior -> MAP -> Optimization 흐름 자동화가 되지 않음.

문제 풀이 측면에서 90% 정도는 같은 패턴이다. 예를들어 시험 문제 스타일:

• Poisson likelihood
• Gamma prior
• MAP estimate \(\lambda\)

이때의 문제 풀이 흐름은 항상 동일하다.

Step 1: Posterior 정의

\[p(\lambda \mid x) \propto p(x \mid \lambda)\, p(\lambda)\]

Step 2: log 취하기

\[\log p(\lambda \mid x) \propto \log p(x \mid \lambda) + \log p(\lambda)\]

Step 3: $\lambda$에 대한 optimization

\[\lambda_{\text{MAP}} = \arg\max_{\lambda} \log p(\lambda \mid x)\]

Step 4: 미분해서 0 되는 지점 찾기

2. likelihood vs posterior 역할 구분이 불분명

왜 \(p(x \lvert z)\) 쓰는지, 왜 \(p(z \lvert x)\) 를 직접 안 쓰는지

중요한 포인트

generative model

모델은 항상 이렇게 정의된다.

\[p(x, z) = p(x \lvert z)p(z)\]

그리고 보통 문제에서 원하는 것은 \(p(z \lvert x)\) 인데 이걸 직접 구하기는 어려움. 왜냐면 이 의미는 posterior, 즉 데이터를 관찰하고 난 후, 즉 실제 뭔가 일이 일어나고 이케이케 조사해보니까 이렇더라. 에 해당하는 거고, 이걸 예측하기 위해 likelihood (데이터 관찰 전 확률) 과 prior (선행 지식)으로 비율 을 보자는것. 그러니까 \(z\) 로 시작하는거를 구하는걸로,,,,,,

3. distribution algebra

예를들어 Poisson likelihhod:

\[p(x \mid \lambda) = frac{\lambda^x e ^{-\lambda}}{x!}\]

log 취하면

\[log p(x \lvert \lambda) = x log \lambda - \lambda - log x!\]

마찬가지로 log를 빨리빨리 못하는 이슈,,,

Distribution algebra cheat sheet (cards)

Poisson likelihood + Gamma prior

• Likelihood

\[p(x\mid\lambda)=\frac{\lambda^x e^{-\lambda}}{x!}\]

• Prior

\[p(\lambda)=\frac{b^a}{\Gamma(a)}\lambda^{a-1}e^{-b\lambda}\]

• Posterior (unnormalized)

\[p(\lambda\mid x)\propto \lambda^{x+a-1}e^{-(b+1)\lambda}\]

• Log form (constants ignored)

\[(x+a-1)\log\lambda-(b+1)\lambda\]

Exponential likelihood + Gamma prior

• Likelihood

\[p(x\mid\lambda)=\lambda e^{-\lambda x}\]

• Prior

\[p(\lambda)\propto \lambda^{a-1}e^{-b\lambda}\]

• Posterior (unnormalized)

\[p(\lambda\mid x)\propto \lambda^{a}e^{-(b+x)\lambda}\]

• Log form (constants ignored)

\[a\log\lambda-(b+x)\lambda\]

Bernoulli likelihood + Beta prior

• Likelihood

\[p(x\mid\theta)=\theta^x(1-\theta)^{1-x}\]

• Prior

\[p(\theta)=\theta^{\alpha-1}(1-\theta)^{\beta-1}\]

• Posterior (unnormalized)

\[p(\theta\mid x)\propto \theta^{x+\alpha-1}(1-\theta)^{\beta-x}\]

• Log form (constants ignored)

\[(x+\alpha-1)\log\theta+(\beta-x)\log(1-\theta)\]

Gaussian likelihood + Gaussian prior (known (\sigma))

• Likelihood

\[p(x\mid\mu)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\]

• Prior

\[p(\mu)=\frac{1}{\sqrt{2\pi\sigma_0^2}}\exp\!\left(-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right)\]

• Posterior (unnormalized)

\[p(\mu\mid x)\propto \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right)\]

• Log form (constants ignored)

\[-\frac{(x-\mu)^2}{2\sigma^2}-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\]

Uniform likelihood + Beta prior on (b)

• Likelihood

\[x\sim U(0,b)\]

• Prior

\[p(b)=\frac{1}{B(\alpha,\beta)}\, b^{\alpha-1}(1-b)^{\beta-1}\]

• Posterior (unnormalized)

\[p(b\mid x)\propto b^{\alpha-1-N}\quad (\text{for } b \ge \max x_i)\]

• Log form (constants ignored)

\[(\alpha-1-N)\log b\]

---

Distribution algebra cheat sheet (2-column quick scan)

• Poisson + Gamma
  • Likelihood: \(p(x\mid\lambda)=\frac{\lambda^x e^{-\lambda}}{x!}\)
  • Prior: \(p(\lambda)=\frac{b^a}{\Gamma(a)}\lambda^{a-1}e^{-b\lambda}\)
  • Posterior (unnorm): \(p(\lambda\mid x)\propto \lambda^{x+a-1}e^{-(b+1)\lambda}\)
  • Log: \((x+a-1)\log\lambda-(b+1)\lambda\)
• Exponential + Gamma
  • Likelihood: \(p(x\mid\lambda)=\lambda e^{-\lambda x}\)
  • Prior: \(p(\lambda)\propto \lambda^{a-1}e^{-b\lambda}\)
  • Posterior (unnorm): \(p(\lambda\mid x)\propto \lambda^{a}e^{-(b+x)\lambda}\)
  • Log: \(a\log\lambda-(b+x)\lambda\)
• Bernoulli + Beta
  • Likelihood: \(p(x\mid\theta)=\theta^x(1-\theta)^{1-x}\)
  • Prior: \(p(\theta)=\theta^{\alpha-1}(1-\theta)^{\beta-1}\)
  • Posterior (unnorm): \(p(\theta\mid x)\propto \theta^{x+\alpha-1}(1-\theta)^{\beta-x}\)
  • Log: \((x+\alpha-1)\log\theta+(\beta-x)\log(1-\theta)\)

  </div>
• Gaussian + Gaussian prior (known \(\sigma\))
  • Likelihood: \(p(x\mid\mu)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\)
  • Prior: \(p(\mu)=\frac{1}{\sqrt{2\pi\sigma_0^2}}\exp\!\left(-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right)\)
  • Posterior (unnorm): \(p(\mu\mid x)\propto \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right)\)
  • Log: \(-\frac{(x-\mu)^2}{2\sigma^2}-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\)
• Uniform + Beta on (b)
  • Likelihood: \(x\sim U(0,b)\)
  • Prior: \(p(b)=\frac{1}{B(\alpha,\beta)}\, b^{\alpha-1}(1-b)^{\beta-1}\)
  • Posterior (unnorm): \(p(b\mid x)\propto b^{\alpha-1-N}\quad (\text{for } b \ge \max x_i)\)
  • Log: \((\alpha-1-N)\log b\)

  </div>

---